From aa28d214089d6b70ab4a49864c234193ddd018cd Mon Sep 17 00:00:00 2001 From: Nissan Pow Date: Wed, 11 Jan 2023 21:45:11 -0800 Subject: [PATCH] Rebaseline from master (#2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Don't explicitly break py2 support (#962) * Don't explicitly break py2 support * Typo * Typo * Pass the paths used by the external interpreter to the one launching the Env Escape server (#960) We were currently not reflecting possible values passed using PYTHONPATH or programmatically added to the interpreter launching the environment escape server. * Bump follow-redirects in /metaflow/plugins/cards/ui (#966) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.7...v1.14.9) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Adding export of graph to JSON for cli (#955) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS (#972) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS * Ran black * emit app tag for AWS Batch jobs (#970) * Bump to 2.5.3 (#974) * Extension packaging improvements (#959) * Fixed dual distribution import in some cases; fixed get_plugin_cli extension If sys.path contains the same directory multiple times, metadata.distributions() will list a package multiple times which breaks the import mechanism we have. This is addressed by skipping duplicate packages. get_plugin_cli for extensions wasn't evaluated late enough therefore not allowing additional parameters to be added. * Fix issue with metaflow_extensions and Conda environment In a Conda environment, it is not possible to re-resolve all the metaflow_extension packages since they appear as a single directory. In that case, we resolve on the INFO file to give us the proper information. Also fixed the way metaflow_extensions are added to the Conda environment to avoid leaking more information than needed. * Validate configuration files for extensions and de-duplicate for internal ns packages * Update packaging of metaflow_extensions packages Packaging is now handled directly by extension_support.py and each distribution/package can define its own suffixes to include. * Rework the import mechanism for modules contained in metaflow_extensions This change does away with the deprecated load_module and also improves handling of loading children modules and _orig modules * Address comments * Squash merge of origin/master * Fix issue when packages were left out * Merge get_pinned_conda_libs * Fix issue with __init__.py in non-distribution packages * Moving card import changes on top of Romain's branch (#973) * Moving card import changes on top of Romain's branch - tiny bug fix to extensions. - Added card related refactor to support mfextinit and regular package. - removing card packaging related code in decorator + everywhere else. * Tiny bug fix on top of romain's new changes. * removing unneccessary logic * Added a bit more inline documentation and addressed documentation comments * Properly handle parsing of package requirements * Fix Pylint errors in some case of metaflow_extensions module aliasing * Properly handle case of files at the root of PYTHONPATH package * Tests for Extensions (#978) * Added tests for extensions. Checking if they work. * dummy commit to see if things work. * fix * debug * bug fix. * possible fix. * tweeking context * Bug fix to tests. * dummy commit * bug fix * Added extension test to core tests. - remove seperate gh action * removing files. * fix * added extension test to py3 context - remove redundant complexity. Co-authored-by: Valay Dave * pass DEFAULT_AWS_CLIENT_PROVIDER to remote tasks (#982) * Fixing some hacky plumbing in card test suite (#967) * Fixing some hacky plumbing for card tests. - Added `--file` arg to card list cli - Using files to get the information written by cli - changes to test to use files instead of stdout - Piping stderrors to stderrr to capture card not found errors. * Removing `\n` from all card tests assertions - `\n` was there because earlier we read stdout. - Now since we read files, it will not be needed. * Simplify mflog (#979) * Fix extension root determination in some cases + card tweaks (#989) * Use importlib_metadata 2.8.3 for Python 3.4, 3.5 but 4.8.3 for Python 3.6+ (#988) * Use importlib_metadata 2.8.3 for Python 3.4 and 3.5 but 4.8.3 for Python 3.6+ This is required because in importlib_metadata 3.4.0, a field called `_normalized_name` was introduced and is relied on by later versions of importlib_metadata. When importlib_metadata looks for distributions, it queries all registered/installed importlib_metadata and if an older version (like 2.8.3) returns something that does not have this field, it causes a crash. * Add __init__.py files to make the vendored package non namespace * Allow configuring the root directory where artifacts go when pulling from S3. (#991) Co-authored-by: kgullikson * Add an option to pass a `role` argument to S3() to provide a role (#987) * Add an option to pass a `role` argument to S3() to provide a role for the S3 client * Move to partial to be able to pickle ops * Address comments * bump version to 2.5.4 (#993) * Bug fix for datetime index in default card (#981) * Bug fix for datetime_indexes. * ran black * Bump minimist from 1.2.5 to 1.2.6 in /metaflow/plugins/cards/ui (#995) Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6. - [Release notes](https://github.com/substack/minimist/releases) - [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6) --- updated-dependencies: - dependency-name: minimist dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix black - set upper bound for click version (#1006) * Add tags to `current` singleton (#1019) * first attempt * fix _set_env param typo * add assertions for tags to current singleton test * Dispatch Metaflow flows to Argo Workflows (#992) * Dispatch Metaflow flows to Argo Workflows * Remove spurious files * Make black happy * Make black happy again * Remove spurious TODOs * Handle non-numeric task ids * Fix typos * don't actively automount service account tokens * make black happy * Update @kubernetes * remove spurious commits * fix issue with hyphens * raise memory for tests * change limits to requests * pin back test resources * update refresh timeout * stylistic nits * Add lint check * make black happy * support gpus * drop extra bracket * fix * Skip sleep if not retryting S3 operations (#1001) * Don't sleep if not retrying S3 operations * Remove unused env variable * VSCode is too smart * Bump minor version for release (#1022) * Alternate way of adding metadata to cloned tasks (#1003) * Alternate way of adding metadata to cloned tasks This adds it on the client side * Clean up metadata passing. Also optimize use within the client * Typo * Fix black test * Keep black happy * Added tests * Ignore namespace when getting origin task * Bump cross-fetch from 3.1.4 to 3.1.5 in /metaflow/plugins/cards/ui (#1027) Bumps [cross-fetch](https://github.com/lquixada/cross-fetch) from 3.1.4 to 3.1.5. - [Release notes](https://github.com/lquixada/cross-fetch/releases) - [Commits](https://github.com/lquixada/cross-fetch/compare/v3.1.4...v3.1.5) --- updated-dependencies: - dependency-name: cross-fetch dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed bug from #1023 (#1025) * current.pathspec to return None when used outside Flow (#1033) * current.pathspec to return None when used outside Flow * formatted with black * Fixing bug in the `card list` command (#1044) - echo_always defaulted to stderr :( * Enable MinIO/EMC-ECS as blob stores for Metaflow (#1045) * Enable EMCECS * Enable MinIO & EMC-ECS blob stores for Metaflow * fix InvalidRange issue * [tag-mutation-project] Stash system info in task metadata (#1039) * stash system info as metadata (in addition to system_tags today) * improve * avoid parsing on ":" for user id * remove date - it's not well defined and created_at is available * python_version is in metadata now, and can vary between orig and resume runs * address CR comments * add the comments * Address issues with S3 get and ranges (#1034) * Address issues with S3 get and ranges This addresses two issues: - get and get_many would not properly populate the range_info in S3Object - get_many on different ranges of the same file would return incorrect results * Add one more test (merge with Netflix internal tests) * Put Run() test behind a check * Address comments * Fix _new_task calling bug in LocalMetadataProvider (#1046) * Release 2.6.1 (#1047) * Support default secrets (#1048) * Run id file should be written prior to execution when resuming (#1051) * Run id file should be written prior to execution when resuming * separate run id test so we can exclude from batch * release 2.6.2 (#1052) * Fix instance metadata calls for IMDSV2 (#1053) * bump version to 2.6.3 (#1054) * Small reorder of imports to make things more consistent (#1055) * [tag-mutation-project] Local metadata provider - object read paths to return ancestral Run tags (#1043) * wip * Fix nits * typo fix. (#1068) * This is the Jackie's tag changes with an additional test and a fix. (#1063) * rebase on master * add replace_many as candidate subcommand * updates from some CR suggestions * propagate opt rename * more changes follow CR * improve consistency guarantees of local metadata tag mutations (retries if a race was lost) * write comment * use replace-many logic to back "tag replace" * finalize client API for tag operations * fix retry logic * Add task.tags == run.tags check in test * [tag-mutation-project] Add tag mutation support to ServiceMetadataProvider * fix version * Enable TagMutationTest in batch and k8s * Bogus commit * minor UX fixes on CLI * Fully support CliCheck for TagMutationTest * add tag and tag set validation to local metadata code path Validate tags on run / resume paths too * flat nor flatten on CLI * Logs really do come via stdout - related to CliCheck refactor * argo and step functions - validate tags as soon as possible (before other user messaging) * no capture_output pre 3.7 * Add another test for tag mutation * Properly pass down the metadata provider There were two cases when the metadata provider was not properly passed down to the client: - when calling the client within a flow - when invoking the `tag` command This fixes both issues and also allows the user to introspect the metadata provider and datastore being used for the flow * Do not make public metadata and datastore in current object * Typo Co-authored-by: Jackie Tung Co-authored-by: jackie-ob <104396218+jackie-ob@users.noreply.github.com> * Add two options to resume: the ability to specify a run-id and the ab… (#1059) * Add two options to resume: the ability to specify a run-id and the ability to only clone tasks The scenario for this use is mainly for external schedulers that are capable of "resuming" failed runs by only re-executing the ones that failed or didn't execute the first time. We need a way for Metaflow to "clone" the tasks that are not re-executed. * Small formatting fix * Made clone-only resume possibly re-entrant * Addressed comments * Clarify the reentrant behavior of resume * Improve sidecar message handling (#1057) * Improve sidecar message handling Several changes: - allow side-passing of a static "context" to the sidecar process to keep messages as small as possible - clean up the interfaces and make it clearer what is what (ie: there is an emitter inside the process and then a sidecar that runs outside) - better error checking - better handling of shutdown (sidecars will now properly get a shutdown message giving them the opportunity to clean up -- this may add a bit of latency at the end of the program execution. Also improved message when plugins don't load in case of an issue with Metaflow extensions * Add the possibility to pass additional context to sidecars This allows more information to be provided after the sidecar is created but still maintains small message sizes * Better handling of initial context message for sidecars In the previous implementation, a failure to send the initial context message could crash the flow. This change properly handles that message like any other message (so has no impact on the flow itself). It also implements better retry policies to send the context. Finally, this change also improves error handling: if a message fails to send, in the previous implementation, it was likely that the next successful one would be an invalid json. * Improve handling of corner conditions on message send * Restart sidecar faster when sidecar process dies * Properly send context after sidecar restart * Remove the specific context from monitors/loggers Subclasses can now handle contexts directly in the subclass. Refactoring also cleaned up what a sidecar and a sidecar worker do by refactoring some of the code out. * Typo: fix names * Minor typos for sidecar changes * Clean up sidecars * More cleanups for sidecars * Addressed final comments for sidecars * Small typo in NullEventLogger.send (#1071) Pushing directly per our conversation with Savin * Bump version to 2.7.0 * Change behavior of env escape when module is present (#1072) When a module configured in the env escape was present in the local environment, we would raise an error. We now print a message and use the module present in the environment (not the environment escape). This allows the env escape modules to provide a "fallback" in case the module is not present but to still use the local module if present * Bump version to 2.7.1 (#1073) * Fix an issue with the environment escape server directory (#1074) The environment escape server would launch in the cwd. This could cause issues if, for example, a `metaflow` directory existed in that cwd (because it would then take precedence over the actual metaflow. To remediate this situation, the server is launched in the configuration directory (where we know what there is). Also took care of a quick annoyance of having to create an empty overrides.py file for everything. This is now no longer required * Support session_vars and client_params in S3() (#1069) * Support session_vars and client_params in S3() This allows the user to pass down session variables and client parameters to set when getting a S3 client. * Addressed comments about default configuration * Support M1 Macs (#1077) * changes to @kubernetes for sandbox (#1079) * Changed error to warning in AWS retry (#1081) Co-authored-by: Preetam Joshi * bump to 2.7.2 (#1083) * Fixed s3util test (#1084) * Fixing s3util test caset * added comment Co-authored-by: Preetam Joshi * Bump svelte from 3.46.1 to 3.49.0 in /metaflow/plugins/cards/ui (#1086) Bumps [svelte](https://github.com/sveltejs/svelte) from 3.46.1 to 3.49.0. - [Release notes](https://github.com/sveltejs/svelte/releases) - [Changelog](https://github.com/sveltejs/svelte/blob/master/CHANGELOG.md) - [Commits](https://github.com/sveltejs/svelte/compare/v3.46.1...v3.49.0) --- updated-dependencies: - dependency-name: svelte dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Metadata version check flag (#1088) * Bump terser from 5.10.0 to 5.14.2 in /metaflow/plugins/cards/ui (#1090) Bumps [terser](https://github.com/terser/terser) from 5.10.0 to 5.14.2. - [Release notes](https://github.com/terser/terser/releases) - [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md) - [Commits](https://github.com/terser/terser/commits) --- updated-dependencies: - dependency-name: terser dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix r CI deps (#1092) * fix r CI deps * fix fractional resoure handling for batch (#1089) * bump version (#1093) * Fix docstrings for the API reference (no functional changes!) (#1076) Make the docstring format in public APIs compliant with the new API reference framework * Move a sys.path modification in s3op to __main__ (#1095) In its current location, this could modify the sys.path of the current running metaflow which could have nefarious consequences with the escape hatch which uses sys.path to determine the outside environment's python path. The following scenario would cause issues: - metaflow is installed in the usual path on the system - a conda environment was manually bootstrapped from a directory A. + at this point, sys.path starts with `A` and then contains the other system includes + s3op.py is imported at some point by the Conda installer when it calls `get_many` + this modifies sys.path to insert, at the beginning, the parent of Metaflow; so in this case, sys.path looks something like ['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...] + when the escape hatch trampolines are created, this sys.path is used to determine what the sys.path for the outside interpreter is. + in A, we create: * INFO * metaflow * metaflow_extensions which properly describe the installation of metaflow - when the escape hatch client runs, it runs in the conda environment and uses metaflow created in A. - when the client wants to start the server, this is where we run into issues because, at this point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages' in which it will find metaflow. It will therefore use that metaflow (which is the same as the one linked in A) to start the server. This runs into issues though because A is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`. This will cause issues if multiple extensions are installed there (it will complain about duplicate configurations for example. The `INFO` file typically used to solve this problem is not read as it was not present for the TL metaflow. This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting sys.path when the module is simply included (and not called as a script). * Airflow Support (#1094) * Airflow on Kubernetes minus Foreachs. - Support for all metaflow construct without foreach and sensors Squashed commit of the following: commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1 Author: Valay Dave Date: Fri Jul 29 01:06:26 2022 +0000 Removed sernsors and banned foreach's commit 8d517c4fecc6568777ad03eca81aaacfa3e91156 Author: Valay Dave Date: Fri Jul 29 00:59:01 2022 +0000 commiting k8s related file from master. commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4 Author: Valay Dave Date: Fri Jul 29 00:54:45 2022 +0000 Uncommented code for foreach support with k8s KubernetesPodOperator version 4.2.0 renamed `resources` to `container_resources` - Check : (https://github.com/apache/airflow/pull/24673) / - (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85) This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator` and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. If it is > 4.2.0 then we set the argument as `container_resources` The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future. So we will only use it for `KuberentesPodOperator` version < 4.2.0 The `resources` argument will also not work for foreach's. commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660 Author: Valay Dave Date: Mon Jul 18 18:31:58 2022 +0000 nit fixes : - fixing comments. - refactor some variable/function names. commit 2079293fbba0d3d862476a7d67b36af8a3389342 Author: Valay Dave Date: Mon Jul 18 18:14:53 2022 +0000 change `token` to `production_token` commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a Author: Valay Dave Date: Mon Jul 18 18:11:56 2022 +0000 Refactored import Airflow Sensors. commit b1472d5f7a629024ca45e8b83700400d02a4d455 Author: Valay Dave Date: Mon Jul 18 18:08:41 2022 +0000 new comment on `startup_timeout_seconds` env var. commit 6d81b758e8f06911258d26f790029f557488a0d7 Author: Valay Dave Date: Mon Jul 18 18:06:09 2022 +0000 Removing traces of `@airflow_schedule_interval` commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946 Author: Valay Dave Date: Thu Jul 14 12:43:08 2022 -0700 Foreach polish (valayDave/metaflow#62) * Removing unused imports * Added validation logic for airflow version numbers with foreaches * Removed `airflow_schedule_interval` decorator. * Added production/deployment token related changes - Uses s3 as a backend to store the production token - Token used for avoiding nameclashes - token stored via `FlowDatastore` * Graph type validation for airflow foreachs - Airflow foreachs only support single node fanout. - validation invalidates graphs with nested foreachs * Added configuration about startup_timeout. * Added final todo on `resources` argument of k8sOp - added a commented code block - it needs to be uncommented when airflow releasese the patch for the op - Code seems feature complete keeping aside airflow patch commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9 Author: Valay Dave Date: Thu Jul 7 19:33:07 2022 +0000 Removed retries from user-defaults. commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43 Author: Valay Dave Date: Wed Jul 6 16:29:33 2022 +0000 updated pod startup time commit fce2bd263f368dbb78a34ac71f64e13c89277222 Author: Valay Dave Date: Wed Jun 29 18:44:11 2022 +0000 Adding default 1 retry for any airflow worker. commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321 Author: Valay Dave Date: Mon Jun 27 01:22:42 2022 +0000 Airflow Foreach Integration - Simple one node foreach-join support as gaurenteed by airflow - Fixed env variable setting issue - introduced MetaflowKuberentesOperator - Created a new operator to allow smootness in plumbing xcom values - Some todos commit d319fa915c558d82f1d127736ce34d3ae0da521d Author: Valay Dave Date: Fri Jun 24 21:12:09 2022 +0000 simplifying run-id macro. commit 0ffc813b1c4e6ba0103be51520f42d191371741a Author: Valay Dave Date: Fri Jun 24 11:51:42 2022 -0700 Refactored parameter macro settings. (valayDave/metaflow#60) commit a3a495077f34183d706c0edbe56d6213766bf5f6 Author: Valay Dave Date: Fri Jun 24 02:05:57 2022 +0000 added comment on need for `start_date` commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec Author: Valay Dave Date: Tue Jun 21 06:03:56 2022 +0000 Refactored an `id_creator` method. commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc Author: Valay Dave Date: Tue Jun 21 05:52:05 2022 +0000 refactor : -`RUN_ID_LEN` to `RUN_HASH_ID_LEN` - `TASK_ID_LEN` to `TASK_ID_HASH_LEN` commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2 Author: Valay Dave Date: Tue Jun 21 05:48:55 2022 +0000 refactored an error string commit 11458188b6c59d044fca0dd2d1f5024ec84f6488 Author: Valay Dave Date: Mon Jun 20 22:42:36 2022 -0700 addressing savins comments. (#59) - Added many adhoc changes based for some comments. - Integrated secrets and `KUBERNETES_SECRETS` - cleaned up parameter setting - cleaned up setting of scheduling interval - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID` - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate` - Checking if dagfile and flow file are same. - fixing variable names. - checking out `kubernetes_decorator.py` from master (6441ed5) - bug fixing secret setting in airflow. - simplified parameter type parsing logic - refactoring airflow argument parsing code. commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646 Author: Valay Dave Date: Mon Jun 13 14:02:57 2022 -0700 Addressing Final comments. (#57) - Added dag-run timeout. - airflow related scheduling checks in decorator. - Auto naming sensors if no name is provided - Annotations to k8s operators - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`) - annotation bug fix - setting`workflow-timeout` for only scheduled dags commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629 Author: Valay Dave Date: Mon Jun 6 04:50:49 2022 +0000 k8s bug fix commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a Author: Valay Dave Date: Mon Jun 6 04:39:50 2022 +0000 removed un-used function commit 70e285e9a7cfbec71fc293508a62c96f33562a01 Author: Valay Dave Date: Mon Jun 6 04:38:37 2022 +0000 Removed unused `sanitize_label` function commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6 Author: Valay Dave Date: Mon Jun 6 04:37:34 2022 +0000 GPU support added + container naming same as argo commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c Author: Valay Dave Date: Mon Jun 6 04:25:17 2022 +0000 Refactored sensors to different files + bug fix - bug caused due `util.compress_list`. - The function doesn't play nice with strings with variety of characters. - Ensured that exceptions are handled appropriately. - Made new file for each sensor under `airflow.sensors` module. commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673 Author: Valay Dave Date: Sat Jun 4 01:41:49 2022 +0000 ran black. commit 558c82f65b383ed0d61ded6bc80326471e284550 Author: Valay Dave Date: Fri Jun 3 18:32:48 2022 -0700 Moving information from airflow_utils to compiler (#56) - commenting todos to organize unfinished changes. - some environment variables set via`V1EnvVar` - `client.V1ObjectFieldSelector` mapped env vars were not working in json form - Moving k8s operator import into its own function. - env vars moved. commit 9bb5f638792a671164ec95891e97f599e9a3385f Author: Valay Dave Date: Fri Jun 3 18:06:03 2022 +0000 added mising Run-id prefixes to variables. - merged `hash` and `dash_connect` filters. commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59 Author: Valay Dave Date: Fri Jun 3 18:00:22 2022 +0000 nit fix : variable name change. commit 660756f952ebd92ba1e26d7f908b81036c31ff10 Author: Valay Dave Date: Fri Jun 3 17:58:34 2022 +0000 nit fixes to dag.py's templating variables. commit 1202f5bc92f76df52b5957f11c8574cadfa62196 Author: Valay Dave Date: Fri Jun 3 17:56:53 2022 +0000 Fixed defaults passing - Addressed comments for airflow.py commit b9387dd428c1a37f9a3bfe2c72cab475da708c02 Author: Valay Dave Date: Fri Jun 3 17:52:24 2022 +0000 Following Changes: - Refactors setting scheduling interval - refactor dag file creating function - refactored is_active to is_paused_upon_creation - removed catchup commit 054e3f389febc6c447494a1dedb01228f5f5650f Author: Valay Dave Date: Fri Jun 3 17:33:25 2022 +0000 Multiple Changes based on comments: 1. refactored `create_k8s_args` into _to_job 2. Addressed comments for snake casing 3. refactored `attrs` for simplicity. 4. refactored `metaflow_parameters` to `parameters`. 5. Refactored setting of `input_paths` commit d481b2fca7914b6b657a69af407cfe1a894a46dc Author: Valay Dave Date: Fri Jun 3 16:42:24 2022 +0000 Removed Sensor metadata extraction. commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3 Author: Valay Dave Date: Fri Jun 3 16:30:34 2022 +0000 porting savin's comments - next changes : addressing comments. commit 3f2353a647e53bc240e28792769c42a71ea8f8c9 Merge: d370ffb c1ff469 Author: Valay Dave Date: Thu Jul 28 23:52:16 2022 +0000 Merge branch 'master' into airflow commit d370ffb248411ad4675f9d55de709dbd75d3806e Merge: a82f144 e4eb751 Author: Valay Dave Date: Thu Jul 14 19:38:48 2022 +0000 Merge branch 'master' into airflow commit a82f1447b414171fc5611758cb6c12fc692f55f9 Merge: bdb1f0d 6f097e3 Author: Valay Dave Date: Wed Jul 13 00:35:49 2022 +0000 Merge branch 'master' into airflow commit bdb1f0dd248d01318d4a493c75b6f54248c7be64 Merge: 8511215 f9a4968 Author: Valay Dave Date: Wed Jun 29 18:44:51 2022 +0000 Merge branch 'master' into airflow commit 85112158cd352cb7de95a2262c011c6f43d98283 Author: Valay Dave Date: Tue Jun 21 02:53:11 2022 +0000 Bug fix from master merge. commit 90c06f12bb14eda51c6a641766c5f67d6763abaa Merge: 0fb73af 6441ed5 Author: Valay Dave Date: Mon Jun 20 21:20:20 2022 +0000 Merge branch 'master' into airflow commit 0fb73af8af9fca2875261e3bdd305a0daab1b229 Author: Valay Dave Date: Sat Jun 4 00:53:10 2022 +0000 squashing bugs after changes from master. commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d Merge: 7bdf662 ffff49b Author: Valay Dave Date: Sat Jun 4 00:20:38 2022 +0000 Merge branch 'master' into af-mmr commit 7bdf662e14966b929b8369c65d5bd3bbe5741937 Author: Valay Dave Date: Mon May 16 17:42:38 2022 -0700 Airflow sensor api (#3) * Fixed run-id setting - Change gaurentees that multiple dags triggered at same moment have unique run-id * added allow multiple in `Decorator` class * Airflow sensor integration. >> support added for : - ExternalTaskSensor - S3KeySensor - SqlSensor >> sensors allow multiple decorators >> sensors accept those arguments which are supported by airflow * Added `@airflow_schedule_interval` decorator * Fixing bug run-id related in env variable setting. commit 2604a29452e794354cf4c612f48bae7cf45856ee Author: Valay Dave Date: Thu Apr 21 18:26:59 2022 +0000 Addressed comments. commit 584e88b679fed7d6eec8ce564bf3707359170568 Author: Valay Dave Date: Wed Apr 20 03:33:55 2022 +0000 fixed printing bug commit 169ac1535e5567149d94749ddaf70264e882d62c Author: Valay Dave Date: Wed Apr 20 03:30:59 2022 +0000 Option help bug fix. commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5 Author: Valay Dave Date: Wed Apr 20 03:25:54 2022 +0000 variable renamemetaflow_specific_args commit 0c779abcd1d9574878da6de8183461b53e0da366 Merge: d299b13 5a61508 Author: Valay Dave Date: Wed Apr 20 03:23:10 2022 +0000 Merge branch 'airflow-tests' into airflow commit 5a61508e61583b567ef8d3fea04e049d74a6d973 Author: Valay Dave Date: Wed Apr 20 03:22:54 2022 +0000 Removing un-used code / resolved-todos. commit d030830f2543f489a1c4ebd17da1b47942f041d6 Author: Valay Dave Date: Wed Apr 20 03:06:03 2022 +0000 ran black, commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45 Merge: f2cb319 7921d13 Author: Valay Dave Date: Wed Apr 20 03:04:19 2022 +0000 Merge branch 'master' into airflow-tests commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa Merge: f2cb319 7921d13 Author: Valay Dave Date: Wed Apr 20 03:02:37 2022 +0000 Merge branch 'master' into airflow commit f2cb3197725f11520da0d49cbeef8de215c243eb Author: Valay Dave Date: Wed Apr 20 02:54:03 2022 +0000 reverting change. commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4 Author: Valay Dave Date: Wed Apr 20 02:47:41 2022 +0000 3 changes: - Removing s3 dep - remove uesless import - added `deployed_on` in dag file template commit c6afba95f5ec05acf7f33fd3228cffd784556e3b Author: Valay Dave Date: Fri Apr 15 22:50:52 2022 +0000 Fixed passing secrets with kubernetes. commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a Author: Valay Dave Date: Fri Apr 15 22:04:22 2022 +0000 Refactored code . - removed compute/k8s.py - Moved k8s code to airflow_compiler.py - ran isort to airflow_compiler.py commit d1c343dbbffbddbebd2aeda26d6846e595144e0b Author: Valay Dave Date: Fri Apr 15 18:02:25 2022 +0000 Added validations about: - un-supported decorators - foreach Changed where validations are done to not save the package. commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce Author: Valay Dave Date: Fri Apr 15 03:34:26 2022 +0000 Fixing mf log related bug - No double logging on metaflow. commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13 Author: Valay Dave Date: Thu Apr 14 03:10:51 2022 +0000 Removed usless code WRT project decorator. commit 5ad9a3949e351b0ac13f11df13446953932e8ffc Author: Valay Dave Date: Thu Apr 14 03:03:19 2022 +0000 Remove readme. commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771 Author: Valay Dave Date: Thu Apr 14 03:02:38 2022 +0000 Made file path required arguement. commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31 Author: Valay Dave Date: Thu Apr 14 03:01:07 2022 +0000 changed `--is-active`->`--is-paused-upon-creation` - dags are active by default. commit 5b98f937a62ee74de8aed8b0efde5045a28f068b Author: Valay Dave Date: Thu Apr 14 02:55:46 2022 +0000 shortened length of run-id and task-id hashes. commit e53426eaa4b156e8bd70ae7510c2e7c66745d101 Author: Valay Dave Date: Thu Apr 14 02:41:32 2022 +0000 Removing un-used args. commit 72cbbfc7424f9be415c22d9144b16a0953f15295 Author: Valay Dave Date: Thu Apr 14 02:39:59 2022 +0000 Moved exceptions to airflow compiler commit b2970ddaa86c393c8abb7f203f6507c386ecbe00 Author: Valay Dave Date: Thu Apr 14 02:33:02 2022 +0000 Changes based on PR comments: - removed airflow xcom push file , moved to decorator code - removed prefix configuration - nit fixes. commit 9e622bac5a75eb9e7a6594d8fa0e47f076634b44 Author: Valay Dave Date: Mon Apr 11 20:39:00 2022 +0000 Removing un-used code paths + code cleanup commit 7425f62cff2c9128eea785223ddeb40fa2d8f503 Author: Valay Dave Date: Mon Apr 11 19:45:04 2022 +0000 Fixing bug fix in schedule. commit eb775cbadd1d2d2c90f160a95a0f42c8ff0d7f4c Author: Valay Dave Date: Sun Apr 10 02:52:59 2022 +0000 Bug fixes WRT Kubernetes secrets + k8s deployments. - Fixing some error messages. - Added some comments. commit 04c92b92c312a4789d3c1e156f61ef57b08dba9f Author: Valay Dave Date: Sun Apr 10 01:20:53 2022 +0000 Added secrets support. commit 4a0a85dff77327640233767e567aee2b379ac13e Author: Valay Dave Date: Sun Apr 10 00:11:46 2022 +0000 Bug fix. commit af91099c0a30c26b58d58696a3ef697ec49a8503 Author: Valay Dave Date: Sun Apr 10 00:03:34 2022 +0000 bug fix. commit c17f04a253dfe6118e2779db79da9669aa2fcef2 Author: Valay Dave Date: Sat Apr 9 23:55:41 2022 +0000 Bug fix in active defaults. commit 0d372361297857076df6af235d1de7005ac1544a Author: Valay Dave Date: Sat Apr 9 23:54:02 2022 +0000 @project, @schedule, default active dag support. - Added a flag to allow setting dag as active on creation - Airflow compatible schedule interval - Project name fixes. commit 5c97b15cb11b5e8279befc5b14c239463750e9b7 Author: Valay Dave Date: Thu Apr 7 21:15:18 2022 +0000 Max workers and worker pool support. commit 9c973f2f44c3cb3a98e3e63f6e4dcef898bc8bf2 Author: Valay Dave Date: Thu Apr 7 19:34:33 2022 +0000 Adding exceptions for missing features. commit 2a946e2f083a34b4b6ed84c70aebf96b084ee8a2 Author: Valay Dave Date: Mon Mar 28 19:34:11 2022 +0000 2 changes : - removed hacky line - added support to directly throw dags in s3. commit e0772ec1bad473482c6fd19f8c5e8b9845303c0a Author: Valay Dave Date: Wed Mar 23 22:38:20 2022 +0000 fixing bugs in service account setting commit 874b94aeeabc664f12551864eff9d8fdc24dc37b Author: Valay Dave Date: Sun Mar 20 23:49:15 2022 +0000 Added support for Branching with Airflow - remove `next` function in `AirflowTask` - `AirflowTask`s has no knowledge of next tasks. - removed todos + added some todos - Graph construction on airflow side using graph_structure datastructure. - graph_structure comes from`FlowGraph.output_steps()[1]` commit 8e9f649bd8c51171c38a1e5af70a44a85e7009ca Author: Valay Dave Date: Sun Mar 20 02:33:04 2022 +0000 Added hacky line commit fd5db04cf0a81b14efda5eaf40cd9227e2bac0d3 Author: Valay Dave Date: Sun Mar 20 02:06:38 2022 +0000 Removed hacky line. commit 5b23eb7d8446bef71246d853b11edafa93c6ef95 Author: Valay Dave Date: Sun Mar 20 01:44:57 2022 +0000 Added support for Parameters. - Supporting int, str, bool, float, JSONType commit c9378e9b284657357ad2997f2b492bc2f4aaefac Author: Valay Dave Date: Sun Mar 20 00:14:10 2022 +0000 Removed todos + added some validation logic. commit 7250a44e1dea1da3464f6f71d0c5188bd314275a Author: Valay Dave Date: Sat Mar 19 23:45:15 2022 +0000 Fixing logs related change from master. commit d125978619ab666dcf96db330acdca40f41b7114 Merge: 8cdac53 7e210a2 Author: Valay Dave Date: Sat Mar 19 23:42:48 2022 +0000 Merge branch 'master' into aft-mm commit 8cdac53dd32648455e36955badb8e0ef7b95a2b3 Author: Valay Dave Date: Sat Mar 19 23:36:47 2022 +0000 making changes sync with master commit 5a93d9f5198c360b2a84ab13a86496986850953c Author: Valay Dave Date: Sat Mar 19 23:29:47 2022 +0000 Fixed bug when using catch + retry commit 62bc8dff68a6171b3b4222075a8e8ac109f65b4c Author: Valay Dave Date: Sat Mar 19 22:58:37 2022 +0000 Changed retry setting. commit 563a20036a2dfcc48101f680f29d4917d53aa247 Author: Valay Dave Date: Sat Mar 19 22:42:57 2022 +0000 Fixed setting `task_id` : - switch task-id from airflow job is to hash to "runid/stepname" - refactor xcom setting variables - added comments commit e2a1e502221dc603385263c82e2c068b9f055188 Author: Valay Dave Date: Sat Mar 19 17:51:59 2022 +0000 setting retry logic. commit a697b56052210c8f009b68772c902bbf77713202 Author: Valay Dave Date: Thu Mar 17 01:02:11 2022 +0000 Nit fix. commit 68f13beb17c7e73c0dddc142ef2418675a506439 Author: Valay Dave Date: Wed Mar 16 20:46:19 2022 +0000 Added @schedule support + readme commit 57bdde54f9ad2c8fe5513dbdb9fd02394664e234 Author: Valay Dave Date: Tue Mar 15 19:47:06 2022 +0000 Fixed setting run-id / task-id to labels in k8s - Fixed setting run-id has from cli macro - added hashing macro to ensure that jinja template set the correct run-id to k8s labels - commit 3d6c31917297d0be5f9915b13680fc415ddb4421 Author: Valay Dave Date: Tue Mar 15 05:39:04 2022 +0000 Got linear workflows working on airflow. - Still not feature complete as lots of args are still unfilled / lots of unknows - minor tweek in eks to ensure airflow is k8s compatible. - passing state around via xcom-push - HACK : AWS keys are passed in a shady way. : Reverse this soon. commit db074b8012f76d9d85225a4ceddb2cde8fefa0f4 Author: Valay Dave Date: Fri Mar 11 12:34:33 2022 -0800 Tweeks commit a9f0468c4721a2017f1b26eb8edcdd80aaa57203 Author: Valay Dave Date: Tue Mar 1 17:14:47 2022 -0800 some changes based on savin's comments. - Added changes to task datastore for different reason : (todo) Decouple these - Added comments to SFN for reference. - Airflow DAG is no longer dependent on metaflow commit f32d089cd3865927bc7510f24ba3418d859410b6 Author: Valay Dave Date: Wed Feb 23 00:54:17 2022 -0800 First version of dynamic dag compiler. - Not completely finished code - Creates generic .py file a JSON that is parsed to create Airflow DAG. - Currently only boiler plate to make a linear dag but doesn't execute anything. - Unfinished code. commit d2def665a86d6a6622d6076882c1c2d54044e773 Author: Valay Dave Date: Sat Feb 19 14:01:47 2022 -0800 more tweeks. commit b176311f166788cc3dfc93354a0c5045a4e6a3d4 Author: Valay Dave Date: Thu Feb 17 09:04:29 2022 -0800 commit 0 - unfinished code. * Making version compatibility changes. - Minimum support version to 2.2.0 * Task-id macro related logic refactor - Done for better version support. * bug fix: param related task-id setting. * applied black * Reverting `decorators.py` to master * Move sys.path insert earlier in s3op.py (#1098) * Update setup.py (#1099) * Fix for env_escape bug when importing local packages (#1100) * Bump to 2.7.5 (#1102) * Fix another issue with the escape hatch and paths (#1105) * Bump to 2.7.6 (#1106) * add a flag to overwrite config when running metaflow configure sandbox (#1103) * card dev docs tiny fix. (#1108) * Fix an issue with get_cards not respecting a Task's ds-root (#1111) get_cards would not always respect a Task's ds-root leading to cases where a Task has cards but they cannot be accessed because of an invalid path. * Adding support for Azure Blob Storage as a datastore (#1091) * Azure Storage * fix BrokenProcessPool import for older pythons * Fix batch * add azure configure, cannot validate storage account url after all (catch-22 in configure) * rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT * fix indent * clean up kubernetes config cli, add secrets * remove signature * fix fractional resoure handling for batch (#1089) * bump version (#1093) * Fix docstrings for the API reference (no functional changes!) (#1076) Make the docstring format in public APIs compliant with the new API reference framework * Fix issue with get_pinned_conda_libs and metaflow extensions * clean up includefile a bit, remove from_env (not used) * Move a sys.path modification in s3op to __main__ (#1095) In its current location, this could modify the sys.path of the current running metaflow which could have nefarious consequences with the escape hatch which uses sys.path to determine the outside environment's python path. The following scenario would cause issues: - metaflow is installed in the usual path on the system - a conda environment was manually bootstrapped from a directory A. + at this point, sys.path starts with `A` and then contains the other system includes + s3op.py is imported at some point by the Conda installer when it calls `get_many` + this modifies sys.path to insert, at the beginning, the parent of Metaflow; so in this case, sys.path looks something like ['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...] + when the escape hatch trampolines are created, this sys.path is used to determine what the sys.path for the outside interpreter is. + in A, we create: * INFO * metaflow * metaflow_extensions which properly describe the installation of metaflow - when the escape hatch client runs, it runs in the conda environment and uses metaflow created in A. - when the client wants to start the server, this is where we run into issues because, at this point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages' in which it will find metaflow. It will therefore use that metaflow (which is the same as the one linked in A) to start the server. This runs into issues though because A is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`. This will cause issues if multiple extensions are installed there (it will complain about duplicate configurations for example. The `INFO` file typically used to solve this problem is not read as it was not present for the TL metaflow. This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting sys.path when the module is simply included (and not called as a script). * Airflow Support (#1094) * Airflow on Kubernetes minus Foreachs. - Support for all metaflow construct without foreach and sensors Squashed commit of the following: commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1 Author: Valay Dave Date: Fri Jul 29 01:06:26 2022 +0000 Removed sernsors and banned foreach's commit 8d517c4fecc6568777ad03eca81aaacfa3e91156 Author: Valay Dave Date: Fri Jul 29 00:59:01 2022 +0000 commiting k8s related file from master. commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4 Author: Valay Dave Date: Fri Jul 29 00:54:45 2022 +0000 Uncommented code for foreach support with k8s KubernetesPodOperator version 4.2.0 renamed `resources` to `container_resources` - Check : (https://github.com/apache/airflow/pull/24673) / - (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85) This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator` and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. If it is > 4.2.0 then we set the argument as `container_resources` The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future. So we will only use it for `KuberentesPodOperator` version < 4.2.0 The `resources` argument will also not work for foreach's. commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660 Author: Valay Dave Date: Mon Jul 18 18:31:58 2022 +0000 nit fixes : - fixing comments. - refactor some variable/function names. commit 2079293fbba0d3d862476a7d67b36af8a3389342 Author: Valay Dave Date: Mon Jul 18 18:14:53 2022 +0000 change `token` to `production_token` commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a Author: Valay Dave Date: Mon Jul 18 18:11:56 2022 +0000 Refactored import Airflow Sensors. commit b1472d5f7a629024ca45e8b83700400d02a4d455 Author: Valay Dave Date: Mon Jul 18 18:08:41 2022 +0000 new comment on `startup_timeout_seconds` env var. commit 6d81b758e8f06911258d26f790029f557488a0d7 Author: Valay Dave Date: Mon Jul 18 18:06:09 2022 +0000 Removing traces of `@airflow_schedule_interval` commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946 Author: Valay Dave Date: Thu Jul 14 12:43:08 2022 -0700 Foreach polish (valayDave/metaflow#62) * Removing unused imports * Added validation logic for airflow version numbers with foreaches * Removed `airflow_schedule_interval` decorator. * Added production/deployment token related changes - Uses s3 as a backend to store the production token - Token used for avoiding nameclashes - token stored via `FlowDatastore` * Graph type validation for airflow foreachs - Airflow foreachs only support single node fanout. - validation invalidates graphs with nested foreachs * Added configuration about startup_timeout. * Added final todo on `resources` argument of k8sOp - added a commented code block - it needs to be uncommented when airflow releasese the patch for the op - Code seems feature complete keeping aside airflow patch commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9 Author: Valay Dave Date: Thu Jul 7 19:33:07 2022 +0000 Removed retries from user-defaults. commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43 Author: Valay Dave Date: Wed Jul 6 16:29:33 2022 +0000 updated pod startup time commit fce2bd263f368dbb78a34ac71f64e13c89277222 Author: Valay Dave Date: Wed Jun 29 18:44:11 2022 +0000 Adding default 1 retry for any airflow worker. commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321 Author: Valay Dave Date: Mon Jun 27 01:22:42 2022 +0000 Airflow Foreach Integration - Simple one node foreach-join support as gaurenteed by airflow - Fixed env variable setting issue - introduced MetaflowKuberentesOperator - Created a new operator to allow smootness in plumbing xcom values - Some todos commit d319fa915c558d82f1d127736ce34d3ae0da521d Author: Valay Dave Date: Fri Jun 24 21:12:09 2022 +0000 simplifying run-id macro. commit 0ffc813b1c4e6ba0103be51520f42d191371741a Author: Valay Dave Date: Fri Jun 24 11:51:42 2022 -0700 Refactored parameter macro settings. (valayDave/metaflow#60) commit a3a495077f34183d706c0edbe56d6213766bf5f6 Author: Valay Dave Date: Fri Jun 24 02:05:57 2022 +0000 added comment on need for `start_date` commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec Author: Valay Dave Date: Tue Jun 21 06:03:56 2022 +0000 Refactored an `id_creator` method. commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc Author: Valay Dave Date: Tue Jun 21 05:52:05 2022 +0000 refactor : -`RUN_ID_LEN` to `RUN_HASH_ID_LEN` - `TASK_ID_LEN` to `TASK_ID_HASH_LEN` commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2 Author: Valay Dave Date: Tue Jun 21 05:48:55 2022 +0000 refactored an error string commit 11458188b6c59d044fca0dd2d1f5024ec84f6488 Author: Valay Dave Date: Mon Jun 20 22:42:36 2022 -0700 addressing savins comments. (#59) - Added many adhoc changes based for some comments. - Integrated secrets and `KUBERNETES_SECRETS` - cleaned up parameter setting - cleaned up setting of scheduling interval - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID` - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate` - Checking if dagfile and flow file are same. - fixing variable names. - checking out `kubernetes_decorator.py` from master (6441ed5) - bug fixing secret setting in airflow. - simplified parameter type parsing logic - refactoring airflow argument parsing code. commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646 Author: Valay Dave Date: Mon Jun 13 14:02:57 2022 -0700 Addressing Final comments. (#57) - Added dag-run timeout. - airflow related scheduling checks in decorator. - Auto naming sensors if no name is provided - Annotations to k8s operators - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`) - annotation bug fix - setting`workflow-timeout` for only scheduled dags commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629 Author: Valay Dave Date: Mon Jun 6 04:50:49 2022 +0000 k8s bug fix commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a Author: Valay Dave Date: Mon Jun 6 04:39:50 2022 +0000 removed un-used function commit 70e285e9a7cfbec71fc293508a62c96f33562a01 Author: Valay Dave Date: Mon Jun 6 04:38:37 2022 +0000 Removed unused `sanitize_label` function commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6 Author: Valay Dave Date: Mon Jun 6 04:37:34 2022 +0000 GPU support added + container naming same as argo commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c Author: Valay Dave Date: Mon Jun 6 04:25:17 2022 +0000 Refactored sensors to different files + bug fix - bug caused due `util.compress_list`. - The function doesn't play nice with strings with variety of characters. - Ensured that exceptions are handled appropriately. - Made new file for each sensor under `airflow.sensors` module. commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673 Author: Valay Dave Date: Sat Jun 4 01:41:49 2022 +0000 ran black. commit 558c82f65b383ed0d61ded6bc80326471e284550 Author: Valay Dave Date: Fri Jun 3 18:32:48 2022 -0700 Moving information from airflow_utils to compiler (#56) - commenting todos to organize unfinished changes. - some environment variables set via`V1EnvVar` - `client.V1ObjectFieldSelector` mapped env vars were not working in json form - Moving k8s operator import into its own function. - env vars moved. commit 9bb5f638792a671164ec95891e97f599e9a3385f Author: Valay Dave Date: Fri Jun 3 18:06:03 2022 +0000 added mising Run-id prefixes to variables. - merged `hash` and `dash_connect` filters. commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59 Author: Valay Dave Date: Fri Jun 3 18:00:22 2022 +0000 nit fix : variable name change. commit 660756f952ebd92ba1e26d7f908b81036c31ff10 Author: Valay Dave Date: Fri Jun 3 17:58:34 2022 +0000 nit fixes to dag.py's templating variables. commit 1202f5bc92f76df52b5957f11c8574cadfa62196 Author: Valay Dave Date: Fri Jun 3 17:56:53 2022 +0000 Fixed defaults passing - Addressed comments for airflow.py commit b9387dd428c1a37f9a3bfe2c72cab475da708c02 Author: Valay Dave Date: Fri Jun 3 17:52:24 2022 +0000 Following Changes: - Refactors setting scheduling interval - refactor dag file creating function - refactored is_active to is_paused_upon_creation - removed catchup commit 054e3f389febc6c447494a1dedb01228f5f5650f Author: Valay Dave Date: Fri Jun 3 17:33:25 2022 +0000 Multiple Changes based on comments: 1. refactored `create_k8s_args` into _to_job 2. Addressed comments for snake casing 3. refactored `attrs` for simplicity. 4. refactored `metaflow_parameters` to `parameters`. 5. Refactored setting of `input_paths` commit d481b2fca7914b6b657a69af407cfe1a894a46dc Author: Valay Dave Date: Fri Jun 3 16:42:24 2022 +0000 Removed Sensor metadata extraction. commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3 Author: Valay Dave Date: Fri Jun 3 16:30:34 2022 +0000 porting savin's comments - next changes : addressing comments. commit 3f2353a647e53bc240e28792769c42a71ea8f8c9 Merge: d370ffb c1ff469 Author: Valay Dave Date: Thu Jul 28 23:52:16 2022 +0000 Merge branch 'master' into airflow commit d370ffb248411ad4675f9d55de709dbd75d3806e Merge: a82f144 e4eb751 Author: Valay Dave Date: Thu Jul 14 19:38:48 2022 +0000 Merge branch 'master' into airflow commit a82f1447b414171fc5611758cb6c12fc692f55f9 Merge: bdb1f0d 6f097e3 Author: Valay Dave Date: Wed Jul 13 00:35:49 2022 +0000 Merge branch 'master' into airflow commit bdb1f0dd248d01318d4a493c75b6f54248c7be64 Merge: 8511215 f9a4968 Author: Valay Dave Date: Wed Jun 29 18:44:51 2022 +0000 Merge branch 'master' into airflow commit 85112158cd352cb7de95a2262c011c6f43d98283 Author: Valay Dave Date: Tue Jun 21 02:53:11 2022 +0000 Bug fix from master merge. commit 90c06f12bb14eda51c6a641766c5f67d6763abaa Merge: 0fb73af 6441ed5 Author: Valay Dave Date: Mon Jun 20 21:20:20 2022 +0000 Merge branch 'master' into airflow commit 0fb73af8af9fca2875261e3bdd305a0daab1b229 Author: Valay Dave Date: Sat Jun 4 00:53:10 2022 +0000 squashing bugs after changes from master. commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d Merge: 7bdf662 ffff49b Author: Valay Dave Date: Sat Jun 4 00:20:38 2022 +0000 Merge branch 'master' into af-mmr commit 7bdf662e14966b929b8369c65d5bd3bbe5741937 Author: Valay Dave Date: Mon May 16 17:42:38 2022 -0700 Airflow sensor api (#3) * Fixed run-id setting - Change gaurentees that multiple dags triggered at same moment have unique run-id * added allow multiple in `Decorator` class * Airflow sensor integration. >> support added for : - ExternalTaskSensor - S3KeySensor - SqlSensor >> sensors allow multiple decorators >> sensors accept those arguments which are supported by airflow * Added `@airflow_schedule_interval` decorator * Fixing bug run-id related in env variable setting. commit 2604a29452e794354cf4c612f48bae7cf45856ee Author: Valay Dave Date: Thu Apr 21 18:26:59 2022 +0000 Addressed comments. commit 584e88b679fed7d6eec8ce564bf3707359170568 Author: Valay Dave Date: Wed Apr 20 03:33:55 2022 +0000 fixed printing bug commit 169ac1535e5567149d94749ddaf70264e882d62c Author: Valay Dave Date: Wed Apr 20 03:30:59 2022 +0000 Option help bug fix. commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5 Author: Valay Dave Date: Wed Apr 20 03:25:54 2022 +0000 variable renamemetaflow_specific_args commit 0c779abcd1d9574878da6de8183461b53e0da366 Merge: d299b13 5a61508 Author: Valay Dave Date: Wed Apr 20 03:23:10 2022 +0000 Merge branch 'airflow-tests' into airflow commit 5a61508e61583b567ef8d3fea04e049d74a6d973 Author: Valay Dave Date: Wed Apr 20 03:22:54 2022 +0000 Removing un-used code / resolved-todos. commit d030830f2543f489a1c4ebd17da1b47942f041d6 Author: Valay Dave Date: Wed Apr 20 03:06:03 2022 +0000 ran black, commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45 Merge: f2cb319 7921d13 Author: Valay Dave Date: Wed Apr 20 03:04:19 2022 +0000 Merge branch 'master' into airflow-tests commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa Merge: f2cb319 7921d13 Author: Valay Dave Date: Wed Apr 20 03:02:37 2022 +0000 Merge branch 'master' into airflow commit f2cb3197725f11520da0d49cbeef8de215c243eb Author: Valay Dave Date: Wed Apr 20 02:54:03 2022 +0000 reverting change. commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4 Author: Valay Dave Date: Wed Apr 20 02:47:41 2022 +0000 3 changes: - Removing s3 dep - remove uesless import - added `deployed_on` in dag file template commit c6afba95f5ec05acf7f33fd3228cffd784556e3b Author: Valay Dave Date: Fri Apr 15 22:50:52 2022 +0000 Fixed passing secrets with kubernetes. commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a Author: Valay Dave Date: Fri Apr 15 22:04:22 2022 +0000 Refactored code . - removed compute/k8s.py - Moved k8s code to airflow_compiler.py - ran isort to airflow_compiler.py commit d1c343dbbffbddbebd2aeda26d6846e595144e0b Author: Valay Dave Date: Fri Apr 15 18:02:25 2022 +0000 Added validations about: - un-supported decorators - foreach Changed where validations are done to not save the package. commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce Author: Valay Dave Date: Fri Apr 15 03:34:26 2022 +0000 Fixing mf log related bug - No double logging on metaflow. commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13 Author: Valay Dave Date: Thu Apr 14 03:10:51 2022 +0000 Removed usless code WRT project decorator. commit 5ad9a3949e351b0ac13f11df13446953932e8ffc Author: Valay Dave Date: Thu Apr 14 03:03:19 2022 +0000 Remove readme. commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771 Author: Valay Dave Date: Thu Apr 14 03:02:38 2022 +0000 Made file path required arguement. commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31 Author: Valay Dave Date: Thu Apr 14 03:01:07 2022 +0000 changed `--is-active`->`--is-paused-upon-creation` - dags are active by default. commit 5b98f937a62ee74de8aed8b0efde5045a28f068b Author: Valay Dave Date: Thu Apr 14 02:55:46 2022 +0000 shortened length of run-id and task-id hashes. commit e53426eaa4b156e8bd70ae7510c2e7c66745d101 Author: Valay Dave Date: Thu Apr 14 02:41:32 2022 +0000 Removing un-used args. commit 72cbbfc7424f9be415c22d9144b16a0953f15295 Author: Valay Dave Date: Thu Apr 14 02:39:59 2022 +0000 Moved exceptions to airflow compiler commit b2970ddaa86c393c8abb7f203f6507c386ecbe00 Author: Valay Dave Date: Thu Apr 14 02:33:02 2022 +0000 Changes based on PR comments: - removed airflow xcom push file , moved to decorator code - removed prefix configuration - nit fixes. commit 9e622bac5a75eb9e7a6594d8fa0e47f076634b44 Author: Valay Dave Date: Mon Apr 11 20:39:00 2022 +0000 Removing un-used code paths + code cleanup commit 7425f62cff2c9128eea785223ddeb40fa2d8f503 Author: Valay Dave Date: Mon Apr 11 19:45:04 2022 +0000 Fixing bug fix in schedule. commit eb775cbadd1d2d2c90f160a95a0f42c8ff0d7f4c Author: Valay Dave Date: Sun Apr 10 02:52:59 2022 +0000 Bug fixes WRT Kubernetes secrets + k8s deployments. - Fixing some error messages. - Added some comments. commit 04c92b92c312a4789d3c1e156f61ef57b08dba9f Author: Valay Dave Date: Sun Apr 10 01:20:53 2022 +0000 Added secrets support. commit 4a0a85dff77327640233767e567aee2b379ac13e Author: Valay Dave Date: Sun Apr 10 00:11:46 2022 +0000 Bug fix. commit af91099c0a30c26b58d58696a3ef697ec49a8503 Author: Valay Dave Date: Sun Apr 10 00:03:34 2022 +0000 bug fix. commit c17f04a253dfe6118e2779db79da9669aa2fcef2 Author: Valay Dave Date: Sat Apr 9 23:55:41 2022 +0000 Bug fix in active defaults. commit 0d372361297857076df6af235d1de7005ac1544a Author: Valay Dave Date: Sat Apr 9 23:54:02 2022 +0000 @project, @schedule, default active dag support. - Added a flag to allow setting dag as active on creation - Airflow compatible schedule interval - Project name fixes. commit 5c97b15cb11b5e8279befc5b14c239463750e9b7 Author: Valay Dave Date: Thu Apr 7 21:15:18 2022 +0000 Max workers and worker pool support. commit 9c973f2f44c3cb3a98e3e63f6e4dcef898bc8bf2 Author: Valay Dave Date: Thu Apr 7 19:34:33 2022 +0000 Adding exceptions for missing features. commit 2a946e2f083a34b4b6ed84c70aebf96b084ee8a2 Author: Valay Dave Date: Mon Mar 28 19:34:11 2022 +0000 2 changes : - removed hacky line - added support to directly throw dags in s3. commit e0772ec1bad473482c6fd19f8c5e8b9845303c0a Author: Valay Dave Date: Wed Mar 23 22:38:20 2022 +0000 fixing bugs in service account setting commit 874b94aeeabc664f12551864eff9d8fdc24dc37b Author: Valay Dave Date: Sun Mar 20 23:49:15 2022 +0000 Added support for Branching with Airflow - remove `next` function in `AirflowTask` - `AirflowTask`s has no knowledge of next tasks. - removed todos + added some todos - Graph construction on airflow side using graph_structure datastructure. - graph_structure comes from`FlowGraph.output_steps()[1]` commit 8e9f649bd8c51171c38a1e5af70a44a85e7009ca Author: Valay Dave Date: Sun Mar 20 02:33:04 2022 +0000 Added hacky line commit fd5db04cf0a81b14efda5eaf40cd9227e2bac0d3 Author: Valay Dave Date: Sun Mar 20 02:06:38 2022 +0000 Removed hacky line. commit 5b23eb7d8446bef71246d853b11edafa93c6ef95 Author: Valay Dave Date: Sun Mar 20 01:44:57 2022 +0000 Added support for Parameters. - Supporting int, str, bool, float, JSONType commit c9378e9b284657357ad2997f2b492bc2f4aaefac Author: Valay Dave Date: Sun Mar 20 00:14:10 2022 +0000 Removed todos + added some validation logic. commit 7250a44e1dea1da3464f6f71d0c5188bd314275a Author: Valay Dave Date: Sat Mar 19 23:45:15 2022 +0000 Fixing logs related change from master. commit d125978619ab666dcf96db330acdca40f41b7114 Merge: 8cdac53 7e210a2 Author: Valay Dave Date: Sat Mar 19 23:42:48 2022 +0000 Merge branch 'master' into aft-mm commit 8cdac53dd32648455e36955badb8e0ef7b95a2b3 Author: Valay Dave Date: Sat Mar 19 23:36:47 2022 +0000 making changes sync with master commit 5a93d9f5198c360b2a84ab13a86496986850953c Author: Valay Dave Date: Sat Mar 19 23:29:47 2022 +0000 Fixed bug when using catch + retry commit 62bc8dff68a6171b3b4222075a8e8ac109f65b4c Author: Valay Dave Date: Sat Mar 19 22:58:37 2022 +0000 Changed retry setting. commit 563a20036a2dfcc48101f680f29d4917d53aa247 Author: Valay Dave Date: Sat Mar 19 22:42:57 2022 +0000 Fixed setting `task_id` : - switch task-id from airflow job is to hash to "runid/stepname" - refactor xcom setting variables - added comments commit e2a1e502221dc603385263c82e2c068b9f055188 Author: Valay Dave Date: Sat Mar 19 17:51:59 2022 +0000 setting retry logic. commit a697b56052210c8f009b68772c902bbf77713202 Author: Valay Dave Date: Thu Mar 17 01:02:11 2022 +0000 Nit fix. commit 68f13beb17c7e73c0dddc142ef2418675a506439 Author: Valay Dave Date: Wed Mar 16 20:46:19 2022 +0000 Added @schedule support + readme commit 57bdde54f9ad2c8fe5513dbdb9fd02394664e234 Author: Valay Dave Date: Tue Mar 15 19:47:06 2022 +0000 Fixed setting run-id / task-id to labels in k8s - Fixed setting run-id has from cli macro - added hashing macro to ensure that jinja template set the correct run-id to k8s labels - commit 3d6c31917297d0be5f9915b13680fc415ddb4421 Author: Valay Dave Date: Tue Mar 15 05:39:04 2022 +0000 Got linear workflows working on airflow. - Still not feature complete as lots of args are still unfilled / lots of unknows - minor tweek in eks to ensure airflow is k8s compatible. - passing state around via xcom-push - HACK : AWS keys are passed in a shady way. : Reverse this soon. commit db074b8012f76d9d85225a4ceddb2cde8fefa0f4 Author: Valay Dave Date: Fri Mar 11 12:34:33 2022 -0800 Tweeks commit a9f0468c4721a2017f1b26eb8edcdd80aaa57203 Author: Valay Dave Date: Tue Mar 1 17:14:47 2022 -0800 some changes based on savin's comments. - Added changes to task datastore for different reason : (todo) Decouple these - Added comments to SFN for reference. - Airflow DAG is no longer dependent on metaflow commit f32d089cd3865927bc7510f24ba3418d859410b6 Author: Valay Dave Date: Wed Feb 23 00:54:17 2022 -0800 First version of dynamic dag compiler. - Not completely finished code - Creates generic .py file a JSON that is parsed to create Airflow DAG. - Currently only boiler plate to make a linear dag but doesn't execute anything. - Unfinished code. commit d2def665a86d6a6622d6076882c1c2d54044e773 Author: Valay Dave Date: Sat Feb 19 14:01:47 2022 -0800 more tweeks. commit b176311f166788cc3dfc93354a0c5045a4e6a3d4 Author: Valay Dave Date: Thu Feb 17 09:04:29 2022 -0800 commit 0 - unfinished code. * Making version compatibility changes. - Minimum support version to 2.2.0 * Task-id macro related logic refactor - Done for better version support. * bug fix: param related task-id setting. * applied black * Reverting `decorators.py` to master * Move sys.path insert earlier in s3op.py (#1098) * Update setup.py (#1099) * Fix for env_escape bug when importing local packages (#1100) * Black reformat * Bump to 2.7.5 (#1102) * Fix another issue with the escape hatch and paths (#1105) * Bump to 2.7.6 (#1106) * add a flag to overwrite config when running metaflow configure sandbox (#1103) * Azure Storage * fix BrokenProcessPool import for older pythons * Fix batch * add azure configure, cannot validate storage account url after all (catch-22 in configure) * rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT * fix indent * clean up kubernetes config cli, add secrets * remove signature * Fix issue with get_pinned_conda_libs and metaflow extensions * clean up includefile a bit, remove from_env (not used) * Black reformat * Fix order of imports in __init__.py * More black * fix "configure azure" merge conflict Co-authored-by: Oleg Avdeev Co-authored-by: Ville Tuulos Co-authored-by: Romain Cledat Co-authored-by: Romain Co-authored-by: Valay Dave Co-authored-by: Savin Co-authored-by: Shashank Srikanth <108034001+hunsdiecker@users.noreply.github.com> * more robust resource type conversions for aws batch/sfn (#1118) * Update setup.py (#1122) * Support airflow with metaflow on azure (#1127) * Fix issue with S3 invocation for conda bootstrap (#1128) * Fix issue with S3 invocation for conda bootstrap * add comment * apply black * Bump minor version to 2.7.8 for release (#1129) * Fix issue with S3 URLs (#1130) * Patch release - 2.7.9 (#1131) * Card bug fix when task-ids are non-unique (#1126) * Card bug fix when task-ids are non-unique - When task-ids are non-unique the cards don't create expected directory paths - To fix the bug we introduce a `steps/` folder - The new changes will be able to read the older version's cards - The older version of clients wont be able to read new version's cards * infering path from folder structure * fix * Writing cards to both paths (steps/tasks) - Added thorough comment about context of change * added `_HACK_SKIP_CARD_DUALWRITE` config var - turns of double writing for cards. * introduced mf config var to skip double write. * Bump version to 2.7.10 to prepare for release (#1136) * Fix DeprecationWarning on invalid escape sequence (#1133) * fix docstring for MetaflowCode (#1134) * fix cpu value formatting for aws batch/sfn (#1140) * Update setup.py (#1141) * Make plugins.airflow.plumbing a well-formed module (#1148) * bump patch version for release (#1149) * Add `cmd` extension point to allow MF extensions to extend the `metaf… (#1143) * Add `cmd` extension point to allow MF extensions to extend the `metaflow` command Simply add a `cmd` directory in your extension and the usual __init__.py or mfextinit_*.py file containing a function called `get_cmd_clis` which returns a list of CLIs to add. Example: @click.group() @click.pass_context def cli(ctx): pass @cli.group(help="My commands") @click.pass_context def foobar(ctx): pass @cli.command(help="Overrides provided `status` command") @click.pass_context def status(ctx): print("I am overriding the usual status command") @foobar.command(help="Some other command") @click.pass_context def baz(ctx): print("Hi!") def get_cmd_clis(): return [cli] * Minor string change * Fix periodic messages printed at runtime (#1061) * Fix periodic messages printed at runtime * Addressed comments -- also rebased * Merged master branch + addressed comments * Pass datastore_type to validate_environment (#1152) * Remove message introduced in #1061 (#1151) * Remove message introduced in #1061 * More message tweak * Minor log msg fix (cannot subscript a set) (#1159) * Support `kubernetes_conn_id` in Airflow integration (#1153) * added `kuberentes_conn_id` * ran black * comment about config variable. * Use json to dump/load decorator specs. (#1144) * Use json to dump/load decorator specs. This cleaned up a few uses of the decospec where individual plugins were having to parse the decospec. Two other minor changes to improve error message and allow for echoing without a nl * Fix to work with remote systems * Addressed comment of making it simpler for simple types (int, float, str) You no longer have to deal with the fun of quoting things for simple types * Removed testing code * argo use kubernetes client class (#1163) * Rewrite IncludeFile implementation (#1109) * Rewrite IncludeFile implementation This rewrite addresses several issues: - IncludeFile were not properly returned in dump and via the client - Improves logic in create/trigger for step-functions for example - Properly consider the size of the included file when dumping artifacts - default value for IncludeFile can also now be a function The code has been cleaned up overall as well and many more comments added. * Fixed tests and removed debugging messages We can now check for the actual artifact value in the CLI checker which is what we now do. * Addressed comments * Fixing bug with include file for schedulers (#1154) * Minor nits * Simplifying parameter creation (#1155) * Add back Azure support (#1162) * Add back Azure support * add some comments explaining return_missing Co-authored-by: Valay Dave Co-authored-by: jackie-ob <104396218+jackie-ob@users.noreply.github.com> * Add options to make card generation faster in the presence of loggers/monitors (#1167) * Env escape improvements and bug fixes (#1166) * Env escape improvements and bug fixes - Properly handle the case of multiple `overrides` for different escaped libraries (previously, only the first override was considered) - Add datetime.timedelta to the list of simple types transferred - Allow the specification of override functions (and getattr/setattr) on additional proxied types. * More cleanup * Forgot a file * Load overrides as relative module even on server side * Terminate server if EOF reached * Allow figures in `Image.from_matplotlib` (#1147) * Allow figures in `Image.from_matplotlib` * nit fix * Bump for release (#1168) * fix pandas call bug (#1173) pandas imported as pd, so pandas.DataFrame will fail * Metaflow pathspec in Airflow UI (#1119) * changes to allow metaflow pathspec in airflow ui - pathspec accessible in the airflow rendered template section of task instance * added runid to the list of rendered task strings * added more metadata about run in rendered fields * tiny refactor. * Allow the input paths to be passed via a file (#1181) * Check compatibility for R 4.2 (#1160) Test PR - DNR * issue 1040 fix: apply _sanitize to template names in Argo workflows (#1180) * issue 1040 fix: apply _sanitize to template names in Argo workflows * apply _sanitize to foreach step template names in Argo workflows * Bump version for release * Handle aborted Kubernetes workloads. (#1195) Log the message and the exit code. * Bump loader-utils from 3.2.0 to 3.2.1 in /metaflow/plugins/cards/ui (#1194) Bumps [loader-utils](https://github.com/webpack/loader-utils) from 3.2.0 to 3.2.1. - [Release notes](https://github.com/webpack/loader-utils/releases) - [Changelog](https://github.com/webpack/loader-utils/blob/master/CHANGELOG.md) - [Commits](https://github.com/webpack/loader-utils/compare/v3.2.0...v3.2.1) --- updated-dependencies: - dependency-name: loader-utils dependency-type: indirect ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix `._orig` access for submodules for MF extensions (#1174) * Fix `._orig` access for submodules for MF extensions This addresses an issue where submodules like `mymodule._orig.submodule` would not load properly. This will properly load things like `from .foofa import xyz` from a module or sub-module in the original location but currently does not work if the import path is absolute. * Add test that orig module is accessible (#1192) * Black format Co-authored-by: Tom Furmston * update black (#1199) * allow = in deco spec values (#1197) Co-authored-by: Adam Merberg * Typo repair and PEP8 cleanup (#1190) * Fixed typos, spelling, and grammar * Fixed several simple PEP warnings * Reverted changes to _vendor folder * Black formatting * Black linting * Reset _vendor folder after black formatting * Pin GH tests to Ubuntu 20.04 (#1201) * Pin GH tests to Ubuntu 20.04 Ubuntu latest which maps to 22.04 doesn't support Py 3.5/3.6 on GH actions * Update test.yml * Set gpu resources correctly "--with kubernetes" (#1202) * Set gpu resources correctly "--with kubernetes" GPU resources were not getting propagated when used "--with kubernetes". Now, it does. Testing Done: - Reproduced the problem creating a step with a decorator @resources(gpu=1) and running it "--with kubernetes". The pods were scheduled without gpus. - Verified that after the fix, the pods were scheduled with gpus (request and limit). * Incorporate review comments + fix black. * Clean up configuration variables (#1183) * Clean up configuration variables * Fix bug in argo/airflow; update default value of AZURE_STORAGE_WORKLOAD_TYPE * Forgot file * Clean up what values are passed down Now, if a value is the default, it is not propagated by default since it can be reconstructed simply from the code. With this change, we basically propagate (by default) only those values that require some external knowledge (env vars, configuration file) to set. This reduces the set of values propagated to only the ones the user overrides. * Remove propagate flag; change name of SERVICE_URL for batch * Fixed more instances of INTERNAL_SERVICE_URL; fixed comments * GCP datastore implementation (#1135) * GCP * card rendering to tolerate runs not tagged with metaflow_version * patch GCP impl to match new includefile impl * fix validate function bug * fix GS includefile merge * fix GS workload type config * Bump version; remove R tests (#1204) * Bump version; remove R tests * Remove more R test * Deal with transient errors (like SlowDowns) more effectively for S3 (#1186) * Deal with transient errors (like SlowDowns) more effectively for S3 In the previous incantantion, SlowDown errors were treated as a regular error and everything was retried (making things worse). With this change, we continue retrying the operations that were unsuccessful (and only those). Also use a better internal boto retry policy (unless specified by the user). Modified the tests to inject failures to be able to test the functionality. Tests now have a failure injection rate of 0, 10, 50 or 90 percent. * Added more comments * Addressed comments; added more comments in code * Fix/move data files (#1206) * Move files -- no code change * Fixups for datatools and datastores * Pure moving of code around -- no code modification This commit can be safely ignored when reviewing code; it has no semantic change although it clearly does not work. * Modify CLI commands to make functional * Bump version number * Fix regression causing CL tool to not work. (#1209) * Fix regression causing CL tool to not work. Update version for immediate release * Fix tox to keep github actions happy * Bump qs from 6.5.2 to 6.5.3 in /metaflow/plugins/cards/ui (#1208) Bumps [qs](https://github.com/ljharb/qs) from 6.5.2 to 6.5.3. - [Release notes](https://github.com/ljharb/qs/releases) - [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md) - [Commits](https://github.com/ljharb/qs/compare/v6.5.2...v6.5.3) --- updated-dependencies: - dependency-name: qs dependency-type: indirect ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Adds check for tutorials dir and flattens if necessary (#1211) * Fix bug with datastore backend instantiation (#1210) * Version bump * Reduce @environment arg length for step-functions create (#1215) * Add support for Kubernetes tolerations (#1207) * Add support for Kubernetes tolerations * Revert get_docker_registry import * Fix KUBERNETES_TOLERATIONS default value * Update toleration example * Remove KUBERNETES_NODE_SELECTOR in kubernetes_job.py * Fix black code style * Add param doc to KubernetesDecorator * Fix typo * Serialize tolerations in runtime_step_cli * Fix KUBERNETES_TOLERATIONS config * Fix node_selector env var in the kubernetes decorator * JSON loads KUBERNETES_TOLERATIONS in kubernetes_decorator init * Parse node_selector and tolerations in the decorator * Update comment * Validate tolerations object in kubernetes_decorator.py * Use hard coded tolerations attribute_map * Fix black code style * String formatting compatible with python 3.5 * Use V1Toleration.attribute_map to validate tolerations * Fix black lint * Improve error handling * Fix black lint * readded changes (#1205) Co-authored-by: Dan * Fix CVE-2007-4559 (tar.extractall) (#1213) * Fix CVE-2007-4559 (tar.extractall) See #1177 for more details * Address comment * Support .conda packages (#1221) * Support .conda packages * fix black issues Signed-off-by: dependabot[bot] Co-authored-by: Romain Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Valay Dave Co-authored-by: bishax Co-authored-by: Savin Co-authored-by: Oleg Avdeev Co-authored-by: Kevin Gullikson Co-authored-by: kgullikson Co-authored-by: Yun Wu Co-authored-by: sam-watts <35522542+sam-watts@users.noreply.github.com> Co-authored-by: Kevin Smith Co-authored-by: jackie-ob <104396218+jackie-ob@users.noreply.github.com> Co-authored-by: Jackie Tung Co-authored-by: Romain Cledat Co-authored-by: Preetam Joshi Co-authored-by: Preetam Joshi Co-authored-by: mrfalconer Co-authored-by: Ville Tuulos Co-authored-by: Shashank Srikanth <108034001+hunsdiecker@users.noreply.github.com> Co-authored-by: Tommy Brecher Co-authored-by: Maciej (Mike) Balajewicz <102193656+mbalajew@users.noreply.github.com> Co-authored-by: John Parker Co-authored-by: Shri Javadekar Co-authored-by: Tom Furmston Co-authored-by: Adam Merberg Co-authored-by: Adam Merberg Co-authored-by: James Budarz Co-authored-by: ashrielbrian Co-authored-by: Riccardo Bini Co-authored-by: Daniel Corvesor Co-authored-by: Dan --- .github/workflows/test.yml | 14 +- .pre-commit-config.yaml | 5 +- R/inst/tutorials/02-statistics/README.md | 2 +- R/inst/tutorials/02-statistics/stats.Rmd | 2 +- .../tutorials/05-statistics-redux/README.md | 2 +- README.md | 3 +- docs/Environment escape.md | 18 +- docs/cards.md | 22 +- docs/concurrency.md | 6 +- docs/datastore.md | 14 +- metaflow/__init__.py | 27 +- metaflow/_vendor/v3_5/__init__.py | 1 + .../{ => v3_5}/importlib_metadata.LICENSE | 0 .../{ => v3_5}/importlib_metadata/__init__.py | 2 +- .../{ => v3_5}/importlib_metadata/_compat.py | 0 metaflow/_vendor/{ => v3_5}/zipp.LICENSE | 0 metaflow/_vendor/{ => v3_5}/zipp.py | 0 metaflow/_vendor/v3_6/__init__.py | 1 + .../_vendor/v3_6/importlib_metadata.LICENSE | 13 + .../v3_6/importlib_metadata/__init__.py | 1063 ++++++ .../v3_6/importlib_metadata/_adapters.py | 68 + .../v3_6/importlib_metadata/_collections.py | 30 + .../v3_6/importlib_metadata/_compat.py | 71 + .../v3_6/importlib_metadata/_functools.py | 104 + .../v3_6/importlib_metadata/_itertools.py | 73 + .../_vendor/v3_6/importlib_metadata/_meta.py | 48 + .../_vendor/v3_6/importlib_metadata/_text.py | 99 + .../v3_6/importlib_metadata/py.typed} | 0 .../_vendor/v3_6/typing_extensions.LICENSE | 254 ++ metaflow/_vendor/v3_6/typing_extensions.py | 2908 +++++++++++++++++ metaflow/_vendor/v3_6/zipp.LICENSE | 19 + metaflow/_vendor/v3_6/zipp.py | 329 ++ metaflow/_vendor/vendor_any.txt | 1 + .../_vendor/{vendor.txt => vendor_v3_5.txt} | 1 - metaflow/_vendor/vendor_v3_6.txt | 1 + metaflow/cli.py | 143 +- metaflow/cli_args.py | 2 +- metaflow/client/core.py | 528 ++- metaflow/client/filecache.py | 13 +- metaflow/cmd/__init__.py | 0 .../{main_cli.py => cmd/configure_cmd.py} | 614 ++-- metaflow/cmd/main_cli.py | 140 + metaflow/cmd/tutorials_cmd.py | 160 + metaflow/cmd/util.py | 23 + metaflow/current.py | 125 +- metaflow/datastore/__init__.py | 5 - metaflow/datastore/datastore_storage.py | 4 +- metaflow/datastore/flow_datastore.py | 4 +- metaflow/datastore/task_datastore.py | 40 +- metaflow/datatools/s3.py | 1047 ------ metaflow/debug.py | 7 +- metaflow/decorators.py | 64 +- metaflow/event_logger.py | 38 +- metaflow/exception.py | 4 + metaflow/extension_support.py | 872 +++-- metaflow/flowspec.py | 141 +- metaflow/graph.py | 4 +- metaflow/includefile.py | 691 ++-- metaflow/lint.py | 1 + metaflow/metadata/heartbeat.py | 29 +- metaflow/metadata/metadata.py | 206 +- metaflow/metadata/util.py | 4 +- metaflow/metaflow_config.py | 384 ++- metaflow/metaflow_config_funcs.py | 120 + metaflow/metaflow_environment.py | 74 +- metaflow/metaflow_version.py | 6 +- metaflow/mflog/__init__.py | 44 +- metaflow/mflog/mflog.py | 2 +- metaflow/mflog/redirect_streams.py | 54 - metaflow/mflog/save_logs.py | 5 +- metaflow/mflog/save_logs_periodically.py | 11 +- metaflow/monitor.py | 263 +- metaflow/multicore_utils.py | 8 +- metaflow/package.py | 49 +- metaflow/parameters.py | 122 +- metaflow/plugins/__init__.py | 57 +- metaflow/plugins/airflow/__init__.py | 0 metaflow/plugins/airflow/airflow.py | 693 ++++ metaflow/plugins/airflow/airflow_cli.py | 434 +++ metaflow/plugins/airflow/airflow_decorator.py | 66 + metaflow/plugins/airflow/airflow_utils.py | 672 ++++ metaflow/plugins/airflow/dag.py | 9 + metaflow/plugins/airflow/exception.py | 12 + metaflow/plugins/airflow/plumbing/__init__.py | 0 .../airflow/plumbing/set_parameters.py | 21 + metaflow/plugins/argo/__init__.py | 0 metaflow/plugins/argo/argo_client.py | 182 ++ metaflow/plugins/argo/argo_workflows.py | 1404 ++++++++ metaflow/plugins/argo/argo_workflows_cli.py | 513 +++ .../plugins/argo/argo_workflows_decorator.py | 63 + metaflow/plugins/argo/process_input_paths.py | 19 + metaflow/plugins/aws/aws_client.py | 56 +- metaflow/plugins/aws/aws_utils.py | 10 +- metaflow/plugins/aws/batch/batch.py | 44 +- metaflow/plugins/aws/batch/batch_cli.py | 7 +- metaflow/plugins/aws/batch/batch_client.py | 49 +- metaflow/plugins/aws/batch/batch_decorator.py | 77 +- metaflow/plugins/aws/eks/kubernetes.py | 362 -- .../plugins/aws/eks/kubernetes_decorator.py | 257 -- .../aws/step_functions/dynamo_db_client.py | 22 +- .../aws/step_functions/schedule_decorator.py | 18 + .../aws/step_functions/step_functions.py | 55 +- .../aws/step_functions/step_functions_cli.py | 25 +- .../step_functions_decorator.py | 2 +- metaflow/plugins/azure/__init__.py | 0 metaflow/plugins/azure/azure_exceptions.py | 13 + metaflow/plugins/azure/azure_tail.py | 94 + metaflow/plugins/azure/azure_utils.py | 218 ++ .../azure/blob_service_client_factory.py | 171 + metaflow/plugins/azure/includefile_support.py | 123 + metaflow/plugins/cards/card_cli.py | 43 +- metaflow/plugins/cards/card_client.py | 141 +- metaflow/plugins/cards/card_datastore.py | 152 +- metaflow/plugins/cards/card_decorator.py | 93 +- .../plugins/cards/card_modules/__init__.py | 93 +- metaflow/plugins/cards/card_modules/basic.py | 31 +- metaflow/plugins/cards/card_modules/card.py | 42 +- .../cards/card_modules/chevron/renderer.py | 7 +- .../plugins/cards/card_modules/components.py | 271 +- .../card_modules/convert_to_native_type.py | 24 +- .../cards/card_modules/renderer_tools.py | 8 +- .../plugins/cards/component_serializer.py | 73 +- metaflow/plugins/cards/exception.py | 6 +- metaflow/plugins/cards/ui/package.json | 2 +- .../plugins/cards/ui/public/card-example.json | 2 +- metaflow/plugins/cards/ui/src/store.ts | 2 +- metaflow/plugins/cards/ui/yarn.lock | 138 +- metaflow/plugins/catch_decorator.py | 27 +- metaflow/plugins/conda/__init__.py | 46 + metaflow/plugins/conda/batch_bootstrap.py | 61 +- metaflow/plugins/conda/conda.py | 5 +- metaflow/plugins/conda/conda_environment.py | 16 +- .../plugins/conda/conda_flow_decorator.py | 28 +- .../plugins/conda/conda_step_decorator.py | 134 +- metaflow/plugins/datastores/__init__.py | 0 metaflow/plugins/datastores/azure_storage.py | 397 +++ metaflow/plugins/datastores/gs_storage.py | 275 ++ .../datastores}/local_storage.py | 5 +- .../datastores}/s3_storage.py | 18 +- metaflow/{ => plugins}/datatools/__init__.py | 5 +- metaflow/plugins/datatools/local.py | 152 + metaflow/plugins/datatools/s3/__init__.py | 9 + metaflow/plugins/datatools/s3/s3.py | 1645 ++++++++++ .../datatools/s3}/s3op.py | 747 +++-- .../datatools/s3}/s3tail.py | 4 +- .../datatools/s3}/s3util.py | 38 +- metaflow/plugins/debug_logger.py | 27 +- metaflow/plugins/debug_monitor.py | 53 +- metaflow/plugins/env_escape/__init__.py | 67 +- metaflow/plugins/env_escape/client.py | 113 +- metaflow/plugins/env_escape/client_modules.py | 45 +- .../env_escape/communication/channel.py | 2 +- .../communication/socket_bytestream.py | 3 + .../plugins/env_escape/communication/utils.py | 4 +- .../plugins/env_escape/data_transferer.py | 7 +- metaflow/plugins/env_escape/server.py | 49 +- metaflow/plugins/env_escape/stub.py | 4 +- metaflow/plugins/environment_decorator.py | 17 +- metaflow/plugins/frameworks/pytorch.py | 2 +- metaflow/plugins/gcp/__init__.py | 0 metaflow/plugins/gcp/gs_exceptions.py | 5 + .../plugins/gcp/gs_storage_client_factory.py | 21 + metaflow/plugins/gcp/gs_tail.py | 85 + metaflow/plugins/gcp/gs_utils.py | 65 + metaflow/plugins/gcp/includefile_support.py | 108 + metaflow/plugins/kubernetes/__init__.py | 0 metaflow/plugins/kubernetes/kubernetes.py | 348 ++ .../{aws/eks => kubernetes}/kubernetes_cli.py | 80 +- .../plugins/kubernetes/kubernetes_client.py | 57 + .../kubernetes/kubernetes_decorator.py | 396 +++ .../kubernetes_job.py} | 512 ++- metaflow/plugins/metadata/local.py | 300 +- metaflow/plugins/metadata/service.py | 236 +- metaflow/plugins/parallel_decorator.py | 2 +- metaflow/plugins/project_decorator.py | 14 + metaflow/plugins/resources_decorator.py | 31 +- metaflow/plugins/retry_decorator.py | 26 +- metaflow/plugins/storage_executor.py | 164 + metaflow/plugins/tag_cli.py | 531 +++ .../test_unbounded_foreach_decorator.py | 4 +- metaflow/plugins/timeout_decorator.py | 23 +- metaflow/pylint_wrapper.py | 9 + metaflow/runtime.py | 367 ++- metaflow/sidecar.py | 156 - metaflow/sidecar/__init__.py | 3 + metaflow/sidecar/sidecar.py | 31 + metaflow/sidecar/sidecar_messages.py | 34 + metaflow/sidecar/sidecar_subprocess.py | 237 ++ metaflow/sidecar/sidecar_worker.py | 68 + metaflow/sidecar_messages.py | 24 - metaflow/sidecar_worker.py | 61 - metaflow/tagging_util.py | 76 + metaflow/task.py | 62 +- metaflow/tutorials/01-playlist/playlist.py | 2 +- metaflow/tutorials/02-statistics/README.md | 2 +- metaflow/tutorials/02-statistics/stats.ipynb | 2 +- metaflow/tutorials/02-statistics/stats.py | 4 +- .../tutorials/03-playlist-redux/playlist.py | 4 +- metaflow/tutorials/04-playlist-plus/README.md | 4 +- .../tutorials/04-playlist-plus/playlist.py | 4 +- metaflow/tutorials/05-helloaws/README.md | 8 +- .../tutorials/06-statistics-redux/README.md | 6 +- .../tutorials/06-statistics-redux/stats.ipynb | 2 +- metaflow/tutorials/07-worldview/README.md | 4 +- metaflow/tutorials/08-autopilot/README.md | 2 +- metaflow/util.py | 49 +- metaflow/vendor.py | 143 +- setup.py | 4 +- test/README.md | 6 +- test/core/contexts.json | 38 +- .../test_org/plugins/frameworks/__init__.py | 0 .../test_org/plugins/frameworks/pytorch.py | 8 + .../test_org/plugins/mfextinit_test_org.py | 2 +- test/core/metaflow_test/__init__.py | 40 + test/core/metaflow_test/cli_check.py | 122 +- test/core/metaflow_test/formatter.py | 6 +- test/core/metaflow_test/metadata_check.py | 27 +- test/core/tests/basic_include.py | 28 +- test/core/tests/basic_log.py | 8 +- test/core/tests/basic_tags.py | 10 +- test/core/tests/card_default_editable.py | 4 +- .../tests/card_default_editable_customize.py | 2 +- .../tests/card_default_editable_with_id.py | 11 +- test/core/tests/card_error.py | 3 +- test/core/tests/card_extension_test.py | 58 + test/core/tests/card_id_append.py | 4 +- test/core/tests/card_import.py | 4 +- test/core/tests/card_resume.py | 2 +- test/core/tests/card_simple.py | 6 +- test/core/tests/card_timeout.py | 2 +- test/core/tests/catch_retry.py | 3 +- test/core/tests/current_singleton.py | 8 + test/core/tests/extensions.py | 1 + test/core/tests/large_artifact.py | 6 +- test/core/tests/large_mflog.py | 5 +- test/core/tests/resume_end_step.py | 29 + test/core/tests/resume_start_step.py | 9 +- test/core/tests/run_id_file.py | 34 + test/core/tests/tag_catch.py | 6 +- test/core/tests/tag_mutation.py | 114 + test/data/__init__.py | 6 +- test/data/s3/s3_data.py | 51 +- test/data/s3/test_s3.py | 312 +- test/extensions/README.md | 5 + test/extensions/install_packages.sh | 3 + .../packages/card_via_extinit/README.md | 3 + .../plugins/cards/card_a/__init__.py | 15 + .../plugins/cards/card_b/__init__.py | 15 + .../plugins/cards/mfextinit_X.py | 4 + .../packages/card_via_extinit/setup.py | 21 + .../packages/card_via_init/README.md | 3 + .../card_via_init/plugins/cards/__init__.py | 15 + .../packages/card_via_init/setup.py | 21 + .../packages/card_via_ns_subpackage/README.md | 3 + .../plugins/cards/nssubpackage/__init__.py | 15 + .../packages/card_via_ns_subpackage/setup.py | 21 + test/unit/test_compute_resource_attributes.py | 61 +- test/unit/test_k8s_job_name_sanitizer.py | 33 - test/unit/test_k8s_label_sanitizer.py | 28 - test/unit/test_local_metadata_provider.py | 31 + test_runner | 8 +- tox.ini | 1 + 262 files changed, 22207 insertions(+), 5583 deletions(-) create mode 100644 metaflow/_vendor/v3_5/__init__.py rename metaflow/_vendor/{ => v3_5}/importlib_metadata.LICENSE (100%) rename metaflow/_vendor/{ => v3_5}/importlib_metadata/__init__.py (99%) rename metaflow/_vendor/{ => v3_5}/importlib_metadata/_compat.py (100%) rename metaflow/_vendor/{ => v3_5}/zipp.LICENSE (100%) rename metaflow/_vendor/{ => v3_5}/zipp.py (100%) create mode 100644 metaflow/_vendor/v3_6/__init__.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata.LICENSE create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/__init__.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/_adapters.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/_collections.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/_compat.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/_functools.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/_itertools.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/_meta.py create mode 100644 metaflow/_vendor/v3_6/importlib_metadata/_text.py rename metaflow/{plugins/aws/eks/__init__.py => _vendor/v3_6/importlib_metadata/py.typed} (100%) create mode 100644 metaflow/_vendor/v3_6/typing_extensions.LICENSE create mode 100644 metaflow/_vendor/v3_6/typing_extensions.py create mode 100644 metaflow/_vendor/v3_6/zipp.LICENSE create mode 100644 metaflow/_vendor/v3_6/zipp.py create mode 100644 metaflow/_vendor/vendor_any.txt rename metaflow/_vendor/{vendor.txt => vendor_v3_5.txt} (66%) create mode 100644 metaflow/_vendor/vendor_v3_6.txt create mode 100644 metaflow/cmd/__init__.py rename metaflow/{main_cli.py => cmd/configure_cmd.py} (62%) create mode 100644 metaflow/cmd/main_cli.py create mode 100644 metaflow/cmd/tutorials_cmd.py create mode 100644 metaflow/cmd/util.py delete mode 100644 metaflow/datatools/s3.py create mode 100644 metaflow/metaflow_config_funcs.py delete mode 100644 metaflow/mflog/redirect_streams.py create mode 100644 metaflow/plugins/airflow/__init__.py create mode 100644 metaflow/plugins/airflow/airflow.py create mode 100644 metaflow/plugins/airflow/airflow_cli.py create mode 100644 metaflow/plugins/airflow/airflow_decorator.py create mode 100644 metaflow/plugins/airflow/airflow_utils.py create mode 100644 metaflow/plugins/airflow/dag.py create mode 100644 metaflow/plugins/airflow/exception.py create mode 100644 metaflow/plugins/airflow/plumbing/__init__.py create mode 100644 metaflow/plugins/airflow/plumbing/set_parameters.py create mode 100644 metaflow/plugins/argo/__init__.py create mode 100644 metaflow/plugins/argo/argo_client.py create mode 100644 metaflow/plugins/argo/argo_workflows.py create mode 100644 metaflow/plugins/argo/argo_workflows_cli.py create mode 100644 metaflow/plugins/argo/argo_workflows_decorator.py create mode 100644 metaflow/plugins/argo/process_input_paths.py delete mode 100644 metaflow/plugins/aws/eks/kubernetes.py delete mode 100644 metaflow/plugins/aws/eks/kubernetes_decorator.py create mode 100644 metaflow/plugins/azure/__init__.py create mode 100644 metaflow/plugins/azure/azure_exceptions.py create mode 100644 metaflow/plugins/azure/azure_tail.py create mode 100644 metaflow/plugins/azure/azure_utils.py create mode 100644 metaflow/plugins/azure/blob_service_client_factory.py create mode 100644 metaflow/plugins/azure/includefile_support.py create mode 100644 metaflow/plugins/datastores/__init__.py create mode 100644 metaflow/plugins/datastores/azure_storage.py create mode 100644 metaflow/plugins/datastores/gs_storage.py rename metaflow/{datastore => plugins/datastores}/local_storage.py (96%) rename metaflow/{datastore => plugins/datastores}/s3_storage.py (91%) rename metaflow/{ => plugins}/datatools/__init__.py (82%) create mode 100644 metaflow/plugins/datatools/local.py create mode 100644 metaflow/plugins/datatools/s3/__init__.py create mode 100644 metaflow/plugins/datatools/s3/s3.py rename metaflow/{datatools => plugins/datatools/s3}/s3op.py (50%) rename metaflow/{datatools => plugins/datatools/s3}/s3tail.py (89%) rename metaflow/{datatools => plugins/datatools/s3}/s3util.py (63%) create mode 100644 metaflow/plugins/gcp/__init__.py create mode 100644 metaflow/plugins/gcp/gs_exceptions.py create mode 100644 metaflow/plugins/gcp/gs_storage_client_factory.py create mode 100644 metaflow/plugins/gcp/gs_tail.py create mode 100644 metaflow/plugins/gcp/gs_utils.py create mode 100644 metaflow/plugins/gcp/includefile_support.py create mode 100644 metaflow/plugins/kubernetes/__init__.py create mode 100644 metaflow/plugins/kubernetes/kubernetes.py rename metaflow/plugins/{aws/eks => kubernetes}/kubernetes_cli.py (75%) create mode 100644 metaflow/plugins/kubernetes/kubernetes_client.py create mode 100644 metaflow/plugins/kubernetes/kubernetes_decorator.py rename metaflow/plugins/{aws/eks/kubernetes_client.py => kubernetes/kubernetes_job.py} (55%) create mode 100644 metaflow/plugins/storage_executor.py create mode 100644 metaflow/plugins/tag_cli.py delete mode 100644 metaflow/sidecar.py create mode 100644 metaflow/sidecar/__init__.py create mode 100644 metaflow/sidecar/sidecar.py create mode 100644 metaflow/sidecar/sidecar_messages.py create mode 100644 metaflow/sidecar/sidecar_subprocess.py create mode 100644 metaflow/sidecar/sidecar_worker.py delete mode 100644 metaflow/sidecar_messages.py delete mode 100644 metaflow/sidecar_worker.py create mode 100644 metaflow/tagging_util.py create mode 100644 test/core/metaflow_extensions/test_org/plugins/frameworks/__init__.py create mode 100644 test/core/metaflow_extensions/test_org/plugins/frameworks/pytorch.py create mode 100644 test/core/tests/card_extension_test.py create mode 100644 test/core/tests/run_id_file.py create mode 100644 test/core/tests/tag_mutation.py create mode 100644 test/extensions/README.md create mode 100644 test/extensions/install_packages.sh create mode 100644 test/extensions/packages/card_via_extinit/README.md create mode 100644 test/extensions/packages/card_via_extinit/metaflow_extensions/card_via_extinit/plugins/cards/card_a/__init__.py create mode 100644 test/extensions/packages/card_via_extinit/metaflow_extensions/card_via_extinit/plugins/cards/card_b/__init__.py create mode 100644 test/extensions/packages/card_via_extinit/metaflow_extensions/card_via_extinit/plugins/cards/mfextinit_X.py create mode 100644 test/extensions/packages/card_via_extinit/setup.py create mode 100644 test/extensions/packages/card_via_init/README.md create mode 100644 test/extensions/packages/card_via_init/metaflow_extensions/card_via_init/plugins/cards/__init__.py create mode 100644 test/extensions/packages/card_via_init/setup.py create mode 100644 test/extensions/packages/card_via_ns_subpackage/README.md create mode 100644 test/extensions/packages/card_via_ns_subpackage/metaflow_extensions/card_via_ns_subpackage/plugins/cards/nssubpackage/__init__.py create mode 100644 test/extensions/packages/card_via_ns_subpackage/setup.py delete mode 100644 test/unit/test_k8s_job_name_sanitizer.py delete mode 100644 test/unit/test_k8s_label_sanitizer.py create mode 100644 test/unit/test_local_metadata_provider.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 6267603aeea..93a8495edd4 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -10,7 +10,7 @@ on: jobs: pre-commit: - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v2 - uses: actions/setup-python@v2 @@ -22,8 +22,8 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest] - ver: ['3.5', '3.6', '3.7', '3.8', '3.9','3.10',] + os: [ubuntu-20.04, macos-latest] + ver: ['3.5', '3.6', '3.7', '3.8', '3.9', '3.10', '3.11',] steps: - uses: actions/checkout@v2 @@ -47,8 +47,8 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest] - ver: ['4.0', '4.1'] + os: [macos-latest] + ver: ['4.0'] steps: - uses: actions/checkout@v2 @@ -58,8 +58,8 @@ jobs: r-version: ${{ matrix.ver }} - name: Install R ${{ matrix.ver }} system dependencies - if: matrix.os == 'ubuntu-latest' - run: sudo apt-get update; sudo apt-get install -y libcurl4-openssl-dev qpdf libgit2-dev + if: matrix.os == 'ubuntu-20.04' + run: sudo apt-get update; sudo apt-get install -y libcurl4-openssl-dev qpdf libgit2-dev libharfbuzz-dev libfribidi-dev - name: Install R ${{ matrix.ver }} Rlang dependencies run: | diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 0fdd3bae214..648aaa7bf51 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -6,8 +6,9 @@ repos: - id: check-yaml - id: check-json - repo: https://github.com/ambv/black - rev: 21.9b0 + rev: 22.10.0 hooks: - id: black language_version: python3 - exclude: "^metaflow/_vendor/" \ No newline at end of file + exclude: "^metaflow/_vendor/" + additional_dependencies: ["click<8.1.0"] diff --git a/R/inst/tutorials/02-statistics/README.md b/R/inst/tutorials/02-statistics/README.md index e96085d119f..523b74f94c5 100644 --- a/R/inst/tutorials/02-statistics/README.md +++ b/R/inst/tutorials/02-statistics/README.md @@ -1,6 +1,6 @@ # Episode 02-statistics: Is this Data Science? -**Use metaflow to load the movie metadata CSV file into a data frame and compute some movie genre specific statistics. These statistics are then used in +**Use metaflow to load the movie metadata CSV file into a data frame and compute some movie genre-specific statistics. These statistics are then used in later examples to improve our playlist generator. You can optionally use the Metaflow client to eyeball the results in a Markdown Notebook, and make some simple plots.** diff --git a/R/inst/tutorials/02-statistics/stats.Rmd b/R/inst/tutorials/02-statistics/stats.Rmd index 6db6f0345d5..a805e194899 100644 --- a/R/inst/tutorials/02-statistics/stats.Rmd +++ b/R/inst/tutorials/02-statistics/stats.Rmd @@ -5,7 +5,7 @@ output: df_print: paged --- -MovieStatsFlow loads the movie metadata CSV file into a Pandas Dataframe and computes some movie genre specific statistics. You can use this notebook and the Metaflow client to eyeball the results and make some simple plots. +MovieStatsFlow loads the movie metadata CSV file into a Pandas Dataframe and computes some movie genre-specific statistics. You can use this notebook and the Metaflow client to eyeball the results and make some simple plots. ```{r} suppressPackageStartupMessages(library(metaflow)) diff --git a/R/inst/tutorials/05-statistics-redux/README.md b/R/inst/tutorials/05-statistics-redux/README.md index e1873ca9754..22142fcc1c0 100644 --- a/R/inst/tutorials/05-statistics-redux/README.md +++ b/R/inst/tutorials/05-statistics-redux/README.md @@ -6,7 +6,7 @@ running on remote compute. In this example we re-run the 'stats.R' workflow adding the '--with batch' command line argument. This instructs Metaflow to run all your steps on AWS batch without changing any code. You can control the behavior with additional arguments, like '--max-workers'. For this example, -'max-workers' is used to limit the number of parallel genre specific statistics +'max-workers' is used to limit the number of parallel genre-specific statistics computations. You can then access the data artifacts (even the local CSV file) from anywhere because the data is being stored in AWS S3.** diff --git a/README.md b/README.md index edae62e93da..aae74f14b26 100644 --- a/README.md +++ b/README.md @@ -51,4 +51,5 @@ We welcome contributions to Metaflow. Please see our [contribution guide](https: ### Code style -We use [black](https://black.readthedocs.io/en/stable/) as a code formatter. The easiest way to ensure your commits are always formatted with the correct version of `black` it is to use [pre-commit](https://pre-commit.com/): install it and then run `pre-commit install` once in your local copy of the repo. \ No newline at end of file +We use [black](https://black.readthedocs.io/en/stable/) as a code formatter. The easiest way to ensure your commits are always formatted with the correct version of `black` it is to use [pre-commit](https://pre-commit.com/): install it and then run `pre-commit install` once in your local copy of the repo. + diff --git a/docs/Environment escape.md b/docs/Environment escape.md index a4d76e3ecd7..c0b03354872 100644 --- a/docs/Environment escape.md +++ b/docs/Environment escape.md @@ -20,7 +20,7 @@ but *some* can execute in another Python environment. At a high-level, the environment escape plugin allows a Python interpreter to forward calls to another interpreter. To set semantics, we will say that a *client* interpreter escapes to a *server* interpreter. The *server* interpreter -operates in a slave-like mode with regards to the *client*. To give a concrete +operates in a slave-like mode with regard to the *client*. To give a concrete example, imagine a package ``data_accessor`` that is available in the base environment you are executing in but not in your Conda environment. When executing within the Conda environment, the *client* interpreter is the Conda @@ -69,7 +69,7 @@ identifier to find the correct stub. There is therefore a **one-to-one mapping between stub objects on the client and backing objects on the server**. The next method called on ```job``` is ```wait``` which returns ```None```. In -this system, by design, only certain objects are able to be transferred between +this system, by design, only certain objects may be transferred between the client and the server: - any Python basic type; this can be extended to any object that can be pickled without any external library; @@ -224,9 +224,9 @@ everything to the server: performs computations at the request of the client when the client is unable to do so. - The server is thus started by the client and the client is responsible for - terminating it when it dies. A big part of the client and server code consist - in loading the configuration for the emulated module, particularly the + The server is thus started by the client, and the client is responsible for + terminating the server when it dies. A big part of the client and server code + consist in loading the configuration for the emulated module, particularly the overrides. The steps to bringing up the client/server connection are as follows: @@ -274,7 +274,7 @@ used). ## Defining an emulated module -To define an emulated module, you need to create a sub directory in +To define an emulated module, you need to create a subdirectory in ```plugins/env_escape/configurations``` called ```emulate_``` where `````` is the name of the library you want to emulate. It can be a "list" where ```__``` is the list separator; this allows multiple libraries to be @@ -286,9 +286,9 @@ create two files: - ```EXPORTED_CLASSES```: This is a dictionary of dictionary describing the whitelisted classes. The outermost key is either a string or a tuple of strings and corresponds to the "module" name (it doesn't really have to be - the module but the prefix of the full name of the whitelisted class)). The + the module but the prefix of the full name of the whitelisted class). The inner key is a string and corresponds to the suffix of the whitelisted - class. Finally, the value is the class that the class maps to internally. If + class. Finally, the value is the class to which the class maps internally. If the outermost key is a tuple, all strings in that tuple will be considered aliases of one another. - ```EXPORTED_FUNCTIONS```: This is the same structure as @@ -324,7 +324,7 @@ create two files: define how attributes are accessed. Note that this is not restricted to attributes accessed using the ```getattr``` and ```setattr``` functions but any attribute. Both of these functions take as arguments ```stub```, - ```name``` and ```func``` which is the function to call to call the remote + ```name``` and ```func``` which is the function to call in order to call the remote ```getattr``` or ```setattr```. The ```setattr``` version takes an additional ```value``` argument. The remote versions simply take the target object and the name of the attribute (and ```value``` if it is a ```setattr``` override) diff --git a/docs/cards.md b/docs/cards.md index cf3a5d9fe02..5bddab2b729 100644 --- a/docs/cards.md +++ b/docs/cards.md @@ -29,7 +29,7 @@ Metaflow cards can be created by placing an [`@card` decorator](#@card-decorator Since the cards are stored in the datastore we can access them via the `view/get` commands in the [card_cli](#card-cli) or by using the `get_cards` [function](../metaflow/plugins/cards/card_client.py). -Metaflow ships with a [DefaultCard](#defaultcard) which visualizes artifacts, images, and `pandas.Dataframe`s. Metaflow also ships custom components like `Image`, `Table`, `Markdown` etc. These can be added to a card at `Task` runtime. Cards can also be edited from `@step` code using the [current.card](#editing-metaflowcard-from-@step-code) interface. `current.card` helps add `MetaflowCardComponent`s from `@step` code to a `MetaflowCard`. `current.card` offers methods like `current.card.append` or `current.card['myid']` to helps add components to a card. Since there can be many `@card`s over a `@step`, `@card` also comes with an `id` argument. The `id` argument helps disambigaute the card a component goes to when using `current.card`. For example, setting `@card(id='myid')` and calling `current.card['myid'].append(x)` will append `MetaflowCardComponent` `x` to the card with `id='myid'`. +Metaflow ships with a [DefaultCard](#defaultcard) which visualizes artifacts, images, and `pandas.Dataframe`s. Metaflow also ships custom components like `Image`, `Table`, `Markdown` etc. These can be added to a card at `Task` runtime. Cards can also be edited from `@step` code using the [current.card](#editing-metaflowcard-from-@step-code) interface. `current.card` helps add `MetaflowCardComponent`s from `@step` code to a `MetaflowCard`. `current.card` offers methods like `current.card.append` or `current.card['myid']` to helps add components to a card. Since there can be many `@card`s over a `@step`, `@card` also comes with an `id` argument. The `id` argument helps disambiguate the card a component goes to when using `current.card`. For example, setting `@card(id='myid')` and calling `current.card['myid'].append(x)` will append `MetaflowCardComponent` `x` to the card with `id='myid'`. ### `@card` decorator The `@card` [decorator](../metaflow/plugins/cards/card_decorator.py) is implemented by inheriting the `StepDecorator`. The decorator can be placed over `@step` to create an HTML file visualizing information from the task. @@ -75,7 +75,7 @@ if __name__ == "__main__": ### `CardDatastore` -The [CardDatastore](../metaflow/plugins/cards/card_datastore.py) is used by the the [card_cli](#card-cli) and the [metaflow card client](#access-cards-in-notebooks) (`get_cards`). It exposes methods to get metadata about a card and the paths to cards for a `pathspec`. +The [CardDatastore](../metaflow/plugins/cards/card_datastore.py) is used by the [card_cli](#card-cli) and the [metaflow card client](#access-cards-in-notebooks) (`get_cards`). It exposes methods to get metadata about a card and the paths to cards for a `pathspec`. ### Card CLI Methods exposed by the [card_cli](../metaflow/plugins/cards/.card_cli.py). : @@ -142,12 +142,12 @@ class CustomCard(MetaflowCard): The class consists of the `_get_mustache` method that returns [chevron](https://github.com/noahmorrison/chevron) object ( a `mustache` based [templating engine](http://mustache.github.io/mustache.5.html) ). Using the `mustache` templating engine you can rewrite HTML template file. In the above example the `PATH_TO_CUSTOM_HTML` is the file that holds the `mustache` HTML template. #### Attributes -- `type (str)` : The `type` of card. Needs to ensure correct resolution. -- `ALLOW_USER_COMPONENTS (bool)` : Setting this to `True` will make the a card be user editable. More information on user editable cards can be found [here](#editing-metaflowcard-from-@step-code). +- `type (str)` : The `type` of card. Needs to ensure correct resolution. +- `ALLOW_USER_COMPONENTS (bool)` : Setting this to `True` will make the card be user editable. More information on user editable cards can be found [here](#editing-metaflowcard-from-@step-code). #### `__init__` Parameters - `components` `(List[str])`: `components` is a list of `render`ed `MetaflowCardComponent`s created at `@step` runtime. These are passed to the `card create` cli command via a tempfile path in the `--component-file` argument. -- `graph` `(Dict[str,dict])`: The DAG associated to the flow. It is a dictionary of the form `stepname:step_attributes`. `step_attributes` is a dictionary of metadata about a step , `stepname` is the name of the step in the DAG. +- `graph` `(Dict[str,dict])`: The DAG associated to the flow. It is a dictionary of the form `stepname:step_attributes`. `step_attributes` is a dictionary of metadata about a step , `stepname` is the name of the step in the DAG. - `options` `(dict)`: helps control the behavior of individual cards. - For example, the `DefaultCard` supports `options` as dictionary of the form `{"only_repr":True}`. Here setting `only_repr` as `True` will ensure that all artifacts are serialized with `reprlib.repr` function instead of native object serialization. @@ -201,7 +201,7 @@ class CustomCard(MetaflowCard): ``` ### `DefaultCard` -The [DefaultCard](../metaflow/plugins/cards/card_modules/basic.py) is a default card exposed by metaflow. This will be used when the `@card` decorator is called without any `type` argument or called with `type='default'` argument. It will also be the default card used with cli. The card uses a [HTML template](../metaflow/plugins/cards/card_modules/base.html) along with a [JS](../metaflow/plugins/cards/card_modules/main.js) and a [CSS](../metaflow/plugins/cards/card_modules/bundle.css) files. +The [DefaultCard](../metaflow/plugins/cards/card_modules/basic.py) is a default card exposed by metaflow. This will be used when the `@card` decorator is called without any `type` argument or called with `type='default'` argument. It will also be the default card used with cli. The card uses an [HTML template](../metaflow/plugins/cards/card_modules/base.html) along with a [JS](../metaflow/plugins/cards/card_modules/main.js) and a [CSS](../metaflow/plugins/cards/card_modules/bundle.css) files. The [HTML](../metaflow/plugins/cards/card_modules/base.html) is a template which works with [JS](../metaflow/plugins/cards/card_modules/main.js) and [CSS](../metaflow/plugins/cards/card_modules/bundle.css). @@ -229,25 +229,25 @@ The JS and CSS are created after building the JS and CSS from the [cards-ui](../ def train(self): from metaflow.cards import Markdown from metaflow import current - current.card.append(Markdown('# This is present in the blank card with id "a"')) - current.card['a'].append(Markdown('# This is present in the default card')) + current.card['a'].append(Markdown('# This is present in the blank card with id "a"')) + current.card.append(Markdown('# This is present in the default card')) self.t = dict( hi = 1, hello = 2 ) self.next(self.end) ``` -In the above scenario there are two `@card` decorators which are being customized by `current.card`. The `current.card.append`/ `current.card['a'].append` methods only accepts objects which are subclasses of `MetaflowCardComponent`. The `current.card.append`/ `current.card['a'].append` methods only add a component to **one** card. Since there can be many cards for a `@step`, a **default editabled card** is resolved to disambiguate which card has access to the `append`/`extend` methods within the `@step`. A default editable card is a card that will have access to the `current.card.append`/`current.card.extend` methods. `current.card` resolve the default editable card before a `@step` code gets executed. It sets the default editable card once the last `@card` decorator calls the `task_pre_step` callback. In the above case, `current.card.append` will add a `Markdown` component to the card of type `default`. `current.card['a'].append` will add the `Markdown` to the `blank` card whose `id` is `a`. A `MetaflowCard` can be user editable, if `ALLOW_USER_COMPONENTS` is set to `True`. Since cards can be of many types, **some cards can also be non editable by users** (Cards with `ALLOW_USER_COMPONENTS=False`). Those cards won't be eligible to access the `current.card.append`. A non user editable card can be edited through expicitly setting an `id` and accessing it via `current.card['myid'].append` or by looking it up by its type via `current.card.get(type=’pytorch’)`. +In the above scenario there are two `@card` decorators which are being customized by `current.card`. The `current.card.append`/ `current.card['a'].append` methods only accepts objects which are subclasses of `MetaflowCardComponent`. The `current.card.append`/ `current.card['a'].append` methods only add a component to **one** card. Since there can be many cards for a `@step`, a **default editable card** is resolved to disambiguate which card has access to the `append`/`extend` methods within the `@step`. A default editable card is a card that will have access to the `current.card.append`/`current.card.extend` methods. `current.card` resolve the default editable card before a `@step` code gets executed. It sets the default editable card once the last `@card` decorator calls the `task_pre_step` callback. In the above case, `current.card.append` will add a `Markdown` component to the card of type `default`. `current.card['a'].append` will add the `Markdown` to the `blank` card whose `id` is `a`. A `MetaflowCard` can be user editable, if `ALLOW_USER_COMPONENTS` is set to `True`. Since cards can be of many types, **some cards can also be non-editable by users** (Cards with `ALLOW_USER_COMPONENTS=False`). Those cards won't be eligible to access the `current.card.append`. A non-user editable card can be edited through explicitly setting an `id` and accessing it via `current.card['myid'].append` or by looking it up by its type via `current.card.get(type=’pytorch’)`. #### `current.card` (`CardComponentCollector`) The `CardComponentCollector` is the object responsible for resolving a `MetaflowCardComponent` to the card referenced in the `@card` decorator. -Since there can be many cards, `CardComponentCollector` has a `_finalize` function. The `_finalize` function is called once the **last** `@card` decorator calls `task_pre_step`. The `_finalize` function will try to find the **default editable card** from all the `@card` decorators on the `@step`. The default editable card is the card that can access the `current.card.append`/`current.card.extend` methods. If there are multiple editable cards with no `id` then `current.card` will throw warnings when users call `current.card.append`. This is done because `current.card` cannot resolve which card the component belongs. +Since there can be many cards, `CardComponentCollector` has a `_finalize` function. The `_finalize` function is called once the **last** `@card` decorator calls `task_pre_step`. The `_finalize` function will try to find the **default editable card** from all the `@card` decorators on the `@step`. The default editable card is the card that can access the `current.card.append`/`current.card.extend` methods. If there are multiple editable cards with no `id` then `current.card` will throw warnings when users call `current.card.append`. This is done because `current.card` cannot resolve which card the component belongs. The `@card` decorator also exposes another argument called `customize=True`. **Only one `@card` decorator over a `@step` can have `customize=True`**. Since cards can also be added from CLI when running a flow, adding `@card(customize=True)` will set **that particular card** from the decorator as default editable. This means that `current.card.append` will append to the card belonging to `@card` with `customize=True`. If there is more than one `@card` decorator with `customize=True` then `current.card` will throw warnings that `append` won't work. -One important feature of the `current.card` object is that it will not fail. Even when users try to access `current.card.append` with multiple editable cards, we throw warnings but don't fail. `current.card` will also not fail when a user tries to access a card of a non-existing id via `current.card['mycard']`. Since `current.card['mycard']` gives reference to a `list` of `MetaflowCardComponent`s, `current.card` will return a non-referenced `list` when users try to access the dictionary inteface with a non existing id (`current.card['my_non_existant_card']`). +One important feature of the `current.card` object is that it will not fail. Even when users try to access `current.card.append` with multiple editable cards, we throw warnings but don't fail. `current.card` will also not fail when a user tries to access a card of a non-existing id via `current.card['mycard']`. Since `current.card['mycard']` gives reference to a `list` of `MetaflowCardComponent`s, `current.card` will return a non-referenced `list` when users try to access the dictionary interface with a nonexistent id (`current.card['my_non_existant_card']`). Once the `@step` completes execution, every `@card` decorator will call `current.card._serialize` (`CardComponentCollector._serialize`) to get a JSON serializable list of `str`/`dict` objects. The `_serialize` function internally calls all [component's](#metaflowcardcomponent) `render` function. This list is `json.dump`ed to a `tempfile` and passed to the `card create` subprocess where the `MetaflowCard` can use them in the final output. diff --git a/docs/concurrency.md b/docs/concurrency.md index 53f439c9203..ed058830ab5 100644 --- a/docs/concurrency.md +++ b/docs/concurrency.md @@ -29,7 +29,7 @@ Concurrency is practically never needed during the first two phases. We divide the concurrency constructs into two categories: Primary and Secondary. Whenever possible, you should prefer the constructs in -the first category. The patterns are well established and they have +the first category. The patterns are well established and have been used successfully in the core Metaflow modules, `runtime.py` and `task.py`. The constructs in the second category can be used in subprocesses, outside the core code paths in `runtime.py` and `task.py`. @@ -109,7 +109,7 @@ delay, to avoid the parent from blocking. The sidecar subprocess may die for various reasons, in which case messages sent to it by the parent may be lost. To keep communication -essentially non-blocking and fast, there is no blocking acklowdgement of +essentially non-blocking and fast, there is no blocking acknowledgement of successful message processing by the sidecar. Hence the communication is lossy. In this sense, communication with a sidecar is more akin to UDP than TCP. @@ -139,7 +139,7 @@ Use a sidecar if you need a task that runs during scheduling or execution of user code. A sidecar task can not perform any critical operations that must succeed in order for a task or a run to be considered valid. This makes sidecars suitable only for opportunistic, -best effort tasks. +best-effort tasks. ### 3. Data Parallelism diff --git a/docs/datastore.md b/docs/datastore.md index ba74d85336d..4cb7c83c39e 100644 --- a/docs/datastore.md +++ b/docs/datastore.md @@ -33,8 +33,8 @@ items to operate on (for example, all the keys to fetch) than to call the same API multiple times with a single key at a time. All APIs are designed with batch processing in mind where it makes sense. -#### Separation of responsabilities -Each class implements few functionalities and we attempted to maximize reuse. +#### Separation of responsibilities +Each class implements few functionalities, and we attempted to maximize reuse. The idea is that this will also help in developing newer implementations going forward and being able to surgically change a few things while keeping most of the code the same. @@ -46,7 +46,7 @@ Before going into the design of the datastore itself, it is worth considering Metaflow considers a datastore to have a `datastore_root` which is the base directory of the datastore. Within that directory, Metaflow will create multiple -sub-directories, one per flow (identified by the name of the flow). Within each +subdirectories, one per flow (identified by the name of the flow). Within each of those directories, Metaflow will create one directory per run as well as a `data` directory which will contain all the artifacts ever produced by that flow. @@ -73,7 +73,7 @@ The datastore has several components (starting at the lowest-level): - a `FlowDataStore` ties everything together. A `FlowDataStore` will include a `ContentAddressedStore` and all the `TaskDataStore`s for all the tasks that are part of the flow. The `FlowDataStore` includes functions to find the - `TaskDataStore` for a given task as well as save and load data directly ( + `TaskDataStore` for a given task as well as to save and load data directly ( this is used primarily for data that is not tied to a single task, for example code packages which are more tied to runs). @@ -111,7 +111,7 @@ additional operations: - transforms the data prior to storing; we currently only compress the data but other operations are possible. -Data is always de-duplicated but you can choose to skip the transformation step +Data is always de-duplicated, but you can choose to skip the transformation step by telling the content address store that the data should be stored `raw` (ie: with no transformation). Note that the de-duplication logic happens *prior* to any transformation (so the transformation itself will not impact the de-duplication @@ -120,7 +120,7 @@ logic). Content stored by the content addressed store is addressable using a `key` which is returned when `save_blobs` is called. `raw` objects can also directly be accessed using a `uri` (also returned by `save_blobs`); the `uri` will point to the location -of the `raw` bytes in the underlying `DataStoreStorage` (so for exmaple a local +of the `raw` bytes in the underlying `DataStoreStorage` (so, for example, a local filesystem path or a S3 path). Objects that are not `raw` do not return a `uri` as they should only be accessed through the content addressed store. @@ -155,7 +155,7 @@ At a high level, the `TaskDataStore` is responsible for: - storing artifacts (functions like `save_artifacts`, `persist` help with this) - storing other metadata about the task execution; this can include logs, general information about the task, user-level metadata and any other information - the user wishes the persist about the task. Functions for this include + the user wishes to persist about the task. Functions for this include `save_logs` and `save_metadata`. Internally, functions like `done` will also store information about the task. diff --git a/metaflow/__init__.py b/metaflow/__init__.py index 9e319b36de3..c306eb3429d 100644 --- a/metaflow/__init__.py +++ b/metaflow/__init__.py @@ -39,7 +39,7 @@ class and related decorators. # More questions? If you have any questions, feel free to post a bug report/question on the -Metaflow Github page. +Metaflow GitHub page. """ import importlib @@ -93,20 +93,29 @@ class and related decorators. tl_package = m.split(".")[1] lazy_load_aliases(alias_submodules(extension_module, tl_package, None)) -from .event_logger import EventLogger +# Utilities +from .multicore_utils import parallel_imap_unordered, parallel_map +from .metaflow_profile import profile + +# current runtime singleton +from .current import current # Flow spec from .flowspec import FlowSpec -from .includefile import IncludeFile + from .parameters import Parameter, JSONTypeClass JSONType = JSONTypeClass() -# current runtime singleton -from .current import current - # data layer -from .datatools import S3 +# For historical reasons, we make metaflow.plugins.datatools accessible as +# metaflow.datatools. S3 is also a tool that has historically been available at the +# TL so keep as is. +lazy_load_aliases({"metaflow.datatools": "metaflow.plugins.datatools"}) +from .plugins.datatools import S3 + +# includefile +from .includefile import IncludeFile # Decorators from .decorators import step, _import_plugin_decorators @@ -134,10 +143,6 @@ class and related decorators. DataArtifact, ) -# Utilities -from .multicore_utils import parallel_imap_unordered, parallel_map -from .metaflow_profile import profile - __version_addl__ = [] _ext_debug("Loading top-level modules") for m in _tl_modules: diff --git a/metaflow/_vendor/v3_5/__init__.py b/metaflow/_vendor/v3_5/__init__.py new file mode 100644 index 00000000000..22ae0c5f40e --- /dev/null +++ b/metaflow/_vendor/v3_5/__init__.py @@ -0,0 +1 @@ +# Empty file \ No newline at end of file diff --git a/metaflow/_vendor/importlib_metadata.LICENSE b/metaflow/_vendor/v3_5/importlib_metadata.LICENSE similarity index 100% rename from metaflow/_vendor/importlib_metadata.LICENSE rename to metaflow/_vendor/v3_5/importlib_metadata.LICENSE diff --git a/metaflow/_vendor/importlib_metadata/__init__.py b/metaflow/_vendor/v3_5/importlib_metadata/__init__.py similarity index 99% rename from metaflow/_vendor/importlib_metadata/__init__.py rename to metaflow/_vendor/v3_5/importlib_metadata/__init__.py index 4e3680aa972..429bfa66c4f 100644 --- a/metaflow/_vendor/importlib_metadata/__init__.py +++ b/metaflow/_vendor/v3_5/importlib_metadata/__init__.py @@ -6,7 +6,7 @@ import abc import csv import sys -from metaflow._vendor import zipp +from metaflow._vendor.v3_5 import zipp import operator import functools import itertools diff --git a/metaflow/_vendor/importlib_metadata/_compat.py b/metaflow/_vendor/v3_5/importlib_metadata/_compat.py similarity index 100% rename from metaflow/_vendor/importlib_metadata/_compat.py rename to metaflow/_vendor/v3_5/importlib_metadata/_compat.py diff --git a/metaflow/_vendor/zipp.LICENSE b/metaflow/_vendor/v3_5/zipp.LICENSE similarity index 100% rename from metaflow/_vendor/zipp.LICENSE rename to metaflow/_vendor/v3_5/zipp.LICENSE diff --git a/metaflow/_vendor/zipp.py b/metaflow/_vendor/v3_5/zipp.py similarity index 100% rename from metaflow/_vendor/zipp.py rename to metaflow/_vendor/v3_5/zipp.py diff --git a/metaflow/_vendor/v3_6/__init__.py b/metaflow/_vendor/v3_6/__init__.py new file mode 100644 index 00000000000..22ae0c5f40e --- /dev/null +++ b/metaflow/_vendor/v3_6/__init__.py @@ -0,0 +1 @@ +# Empty file \ No newline at end of file diff --git a/metaflow/_vendor/v3_6/importlib_metadata.LICENSE b/metaflow/_vendor/v3_6/importlib_metadata.LICENSE new file mode 100644 index 00000000000..be7e092b0b0 --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata.LICENSE @@ -0,0 +1,13 @@ +Copyright 2017-2019 Jason R. Coombs, Barry Warsaw + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/metaflow/_vendor/v3_6/importlib_metadata/__init__.py b/metaflow/_vendor/v3_6/importlib_metadata/__init__.py new file mode 100644 index 00000000000..8d3b7814d50 --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/__init__.py @@ -0,0 +1,1063 @@ +import os +import re +import abc +import csv +import sys +from metaflow._vendor.v3_6 import zipp +import email +import pathlib +import operator +import textwrap +import warnings +import functools +import itertools +import posixpath +import collections + +from . import _adapters, _meta +from ._collections import FreezableDefaultDict, Pair +from ._compat import ( + NullFinder, + install, + pypy_partial, +) +from ._functools import method_cache, pass_none +from ._itertools import always_iterable, unique_everseen +from ._meta import PackageMetadata, SimplePath + +from contextlib import suppress +from importlib import import_module +from importlib.abc import MetaPathFinder +from itertools import starmap +from typing import List, Mapping, Optional, Union + + +__all__ = [ + 'Distribution', + 'DistributionFinder', + 'PackageMetadata', + 'PackageNotFoundError', + 'distribution', + 'distributions', + 'entry_points', + 'files', + 'metadata', + 'packages_distributions', + 'requires', + 'version', +] + + +class PackageNotFoundError(ModuleNotFoundError): + """The package was not found.""" + + def __str__(self): + return f"No package metadata was found for {self.name}" + + @property + def name(self): + (name,) = self.args + return name + + +class Sectioned: + """ + A simple entry point config parser for performance + + >>> for item in Sectioned.read(Sectioned._sample): + ... print(item) + Pair(name='sec1', value='# comments ignored') + Pair(name='sec1', value='a = 1') + Pair(name='sec1', value='b = 2') + Pair(name='sec2', value='a = 2') + + >>> res = Sectioned.section_pairs(Sectioned._sample) + >>> item = next(res) + >>> item.name + 'sec1' + >>> item.value + Pair(name='a', value='1') + >>> item = next(res) + >>> item.value + Pair(name='b', value='2') + >>> item = next(res) + >>> item.name + 'sec2' + >>> item.value + Pair(name='a', value='2') + >>> list(res) + [] + """ + + _sample = textwrap.dedent( + """ + [sec1] + # comments ignored + a = 1 + b = 2 + + [sec2] + a = 2 + """ + ).lstrip() + + @classmethod + def section_pairs(cls, text): + return ( + section._replace(value=Pair.parse(section.value)) + for section in cls.read(text, filter_=cls.valid) + if section.name is not None + ) + + @staticmethod + def read(text, filter_=None): + lines = filter(filter_, map(str.strip, text.splitlines())) + name = None + for value in lines: + section_match = value.startswith('[') and value.endswith(']') + if section_match: + name = value.strip('[]') + continue + yield Pair(name, value) + + @staticmethod + def valid(line): + return line and not line.startswith('#') + + +class DeprecatedTuple: + """ + Provide subscript item access for backward compatibility. + + >>> recwarn = getfixture('recwarn') + >>> ep = EntryPoint(name='name', value='value', group='group') + >>> ep[:] + ('name', 'value', 'group') + >>> ep[0] + 'name' + >>> len(recwarn) + 1 + """ + + _warn = functools.partial( + warnings.warn, + "EntryPoint tuple interface is deprecated. Access members by name.", + DeprecationWarning, + stacklevel=pypy_partial(2), + ) + + def __getitem__(self, item): + self._warn() + return self._key()[item] + + +class EntryPoint(DeprecatedTuple): + """An entry point as defined by Python packaging conventions. + + See `the packaging docs on entry points + `_ + for more information. + """ + + pattern = re.compile( + r'(?P[\w.]+)\s*' + r'(:\s*(?P[\w.]+))?\s*' + r'(?P\[.*\])?\s*$' + ) + """ + A regular expression describing the syntax for an entry point, + which might look like: + + - module + - package.module + - package.module:attribute + - package.module:object.attribute + - package.module:attr [extra1, extra2] + + Other combinations are possible as well. + + The expression is lenient about whitespace around the ':', + following the attr, and following any extras. + """ + + dist: Optional['Distribution'] = None + + def __init__(self, name, value, group): + vars(self).update(name=name, value=value, group=group) + + def load(self): + """Load the entry point from its definition. If only a module + is indicated by the value, return that module. Otherwise, + return the named object. + """ + match = self.pattern.match(self.value) + module = import_module(match.group('module')) + attrs = filter(None, (match.group('attr') or '').split('.')) + return functools.reduce(getattr, attrs, module) + + @property + def module(self): + match = self.pattern.match(self.value) + return match.group('module') + + @property + def attr(self): + match = self.pattern.match(self.value) + return match.group('attr') + + @property + def extras(self): + match = self.pattern.match(self.value) + return list(re.finditer(r'\w+', match.group('extras') or '')) + + def _for(self, dist): + vars(self).update(dist=dist) + return self + + def __iter__(self): + """ + Supply iter so one may construct dicts of EntryPoints by name. + """ + msg = ( + "Construction of dict of EntryPoints is deprecated in " + "favor of EntryPoints." + ) + warnings.warn(msg, DeprecationWarning) + return iter((self.name, self)) + + def matches(self, **params): + attrs = (getattr(self, param) for param in params) + return all(map(operator.eq, params.values(), attrs)) + + def _key(self): + return self.name, self.value, self.group + + def __lt__(self, other): + return self._key() < other._key() + + def __eq__(self, other): + return self._key() == other._key() + + def __setattr__(self, name, value): + raise AttributeError("EntryPoint objects are immutable.") + + def __repr__(self): + return ( + f'EntryPoint(name={self.name!r}, value={self.value!r}, ' + f'group={self.group!r})' + ) + + def __hash__(self): + return hash(self._key()) + + +class DeprecatedList(list): + """ + Allow an otherwise immutable object to implement mutability + for compatibility. + + >>> recwarn = getfixture('recwarn') + >>> dl = DeprecatedList(range(3)) + >>> dl[0] = 1 + >>> dl.append(3) + >>> del dl[3] + >>> dl.reverse() + >>> dl.sort() + >>> dl.extend([4]) + >>> dl.pop(-1) + 4 + >>> dl.remove(1) + >>> dl += [5] + >>> dl + [6] + [1, 2, 5, 6] + >>> dl + (6,) + [1, 2, 5, 6] + >>> dl.insert(0, 0) + >>> dl + [0, 1, 2, 5] + >>> dl == [0, 1, 2, 5] + True + >>> dl == (0, 1, 2, 5) + True + >>> len(recwarn) + 1 + """ + + _warn = functools.partial( + warnings.warn, + "EntryPoints list interface is deprecated. Cast to list if needed.", + DeprecationWarning, + stacklevel=pypy_partial(2), + ) + + def _wrap_deprecated_method(method_name: str): # type: ignore + def wrapped(self, *args, **kwargs): + self._warn() + return getattr(super(), method_name)(*args, **kwargs) + + return wrapped + + for method_name in [ + '__setitem__', + '__delitem__', + 'append', + 'reverse', + 'extend', + 'pop', + 'remove', + '__iadd__', + 'insert', + 'sort', + ]: + locals()[method_name] = _wrap_deprecated_method(method_name) + + def __add__(self, other): + if not isinstance(other, tuple): + self._warn() + other = tuple(other) + return self.__class__(tuple(self) + other) + + def __eq__(self, other): + if not isinstance(other, tuple): + self._warn() + other = tuple(other) + + return tuple(self).__eq__(other) + + +class EntryPoints(DeprecatedList): + """ + An immutable collection of selectable EntryPoint objects. + """ + + __slots__ = () + + def __getitem__(self, name): # -> EntryPoint: + """ + Get the EntryPoint in self matching name. + """ + if isinstance(name, int): + warnings.warn( + "Accessing entry points by index is deprecated. " + "Cast to tuple if needed.", + DeprecationWarning, + stacklevel=2, + ) + return super().__getitem__(name) + try: + return next(iter(self.select(name=name))) + except StopIteration: + raise KeyError(name) + + def select(self, **params): + """ + Select entry points from self that match the + given parameters (typically group and/or name). + """ + return EntryPoints(ep for ep in self if ep.matches(**params)) + + @property + def names(self): + """ + Return the set of all names of all entry points. + """ + return {ep.name for ep in self} + + @property + def groups(self): + """ + Return the set of all groups of all entry points. + + For coverage while SelectableGroups is present. + >>> EntryPoints().groups + set() + """ + return {ep.group for ep in self} + + @classmethod + def _from_text_for(cls, text, dist): + return cls(ep._for(dist) for ep in cls._from_text(text)) + + @staticmethod + def _from_text(text): + return ( + EntryPoint(name=item.value.name, value=item.value.value, group=item.name) + for item in Sectioned.section_pairs(text or '') + ) + + +class Deprecated: + """ + Compatibility add-in for mapping to indicate that + mapping behavior is deprecated. + + >>> recwarn = getfixture('recwarn') + >>> class DeprecatedDict(Deprecated, dict): pass + >>> dd = DeprecatedDict(foo='bar') + >>> dd.get('baz', None) + >>> dd['foo'] + 'bar' + >>> list(dd) + ['foo'] + >>> list(dd.keys()) + ['foo'] + >>> 'foo' in dd + True + >>> list(dd.values()) + ['bar'] + >>> len(recwarn) + 1 + """ + + _warn = functools.partial( + warnings.warn, + "SelectableGroups dict interface is deprecated. Use select.", + DeprecationWarning, + stacklevel=pypy_partial(2), + ) + + def __getitem__(self, name): + self._warn() + return super().__getitem__(name) + + def get(self, name, default=None): + self._warn() + return super().get(name, default) + + def __iter__(self): + self._warn() + return super().__iter__() + + def __contains__(self, *args): + self._warn() + return super().__contains__(*args) + + def keys(self): + self._warn() + return super().keys() + + def values(self): + self._warn() + return super().values() + + +class SelectableGroups(Deprecated, dict): + """ + A backward- and forward-compatible result from + entry_points that fully implements the dict interface. + """ + + @classmethod + def load(cls, eps): + by_group = operator.attrgetter('group') + ordered = sorted(eps, key=by_group) + grouped = itertools.groupby(ordered, by_group) + return cls((group, EntryPoints(eps)) for group, eps in grouped) + + @property + def _all(self): + """ + Reconstruct a list of all entrypoints from the groups. + """ + groups = super(Deprecated, self).values() + return EntryPoints(itertools.chain.from_iterable(groups)) + + @property + def groups(self): + return self._all.groups + + @property + def names(self): + """ + for coverage: + >>> SelectableGroups().names + set() + """ + return self._all.names + + def select(self, **params): + if not params: + return self + return self._all.select(**params) + + +class PackagePath(pathlib.PurePosixPath): + """A reference to a path in a package""" + + def read_text(self, encoding='utf-8'): + with self.locate().open(encoding=encoding) as stream: + return stream.read() + + def read_binary(self): + with self.locate().open('rb') as stream: + return stream.read() + + def locate(self): + """Return a path-like object for this path""" + return self.dist.locate_file(self) + + +class FileHash: + def __init__(self, spec): + self.mode, _, self.value = spec.partition('=') + + def __repr__(self): + return f'' + + +class Distribution: + """A Python distribution package.""" + + @abc.abstractmethod + def read_text(self, filename): + """Attempt to load metadata file given by the name. + + :param filename: The name of the file in the distribution info. + :return: The text if found, otherwise None. + """ + + @abc.abstractmethod + def locate_file(self, path): + """ + Given a path to a file in this distribution, return a path + to it. + """ + + @classmethod + def from_name(cls, name): + """Return the Distribution for the given package name. + + :param name: The name of the distribution package to search for. + :return: The Distribution instance (or subclass thereof) for the named + package, if found. + :raises PackageNotFoundError: When the named package's distribution + metadata cannot be found. + """ + for resolver in cls._discover_resolvers(): + dists = resolver(DistributionFinder.Context(name=name)) + dist = next(iter(dists), None) + if dist is not None: + return dist + else: + raise PackageNotFoundError(name) + + @classmethod + def discover(cls, **kwargs): + """Return an iterable of Distribution objects for all packages. + + Pass a ``context`` or pass keyword arguments for constructing + a context. + + :context: A ``DistributionFinder.Context`` object. + :return: Iterable of Distribution objects for all packages. + """ + context = kwargs.pop('context', None) + if context and kwargs: + raise ValueError("cannot accept context and kwargs") + context = context or DistributionFinder.Context(**kwargs) + return itertools.chain.from_iterable( + resolver(context) for resolver in cls._discover_resolvers() + ) + + @staticmethod + def at(path): + """Return a Distribution for the indicated metadata path + + :param path: a string or path-like object + :return: a concrete Distribution instance for the path + """ + return PathDistribution(pathlib.Path(path)) + + @staticmethod + def _discover_resolvers(): + """Search the meta_path for resolvers.""" + declared = ( + getattr(finder, 'find_distributions', None) for finder in sys.meta_path + ) + return filter(None, declared) + + @classmethod + def _local(cls, root='.'): + from pep517 import build, meta + + system = build.compat_system(root) + builder = functools.partial( + meta.build, + source_dir=root, + system=system, + ) + return PathDistribution(zipp.Path(meta.build_as_zip(builder))) + + @property + def metadata(self) -> _meta.PackageMetadata: + """Return the parsed metadata for this Distribution. + + The returned object will have keys that name the various bits of + metadata. See PEP 566 for details. + """ + text = ( + self.read_text('METADATA') + or self.read_text('PKG-INFO') + # This last clause is here to support old egg-info files. Its + # effect is to just end up using the PathDistribution's self._path + # (which points to the egg-info file) attribute unchanged. + or self.read_text('') + ) + return _adapters.Message(email.message_from_string(text)) + + @property + def name(self): + """Return the 'Name' metadata for the distribution package.""" + return self.metadata['Name'] + + @property + def _normalized_name(self): + """Return a normalized version of the name.""" + return Prepared.normalize(self.name) + + @property + def version(self): + """Return the 'Version' metadata for the distribution package.""" + return self.metadata['Version'] + + @property + def entry_points(self): + return EntryPoints._from_text_for(self.read_text('entry_points.txt'), self) + + @property + def files(self): + """Files in this distribution. + + :return: List of PackagePath for this distribution or None + + Result is `None` if the metadata file that enumerates files + (i.e. RECORD for dist-info or SOURCES.txt for egg-info) is + missing. + Result may be empty if the metadata exists but is empty. + """ + + def make_file(name, hash=None, size_str=None): + result = PackagePath(name) + result.hash = FileHash(hash) if hash else None + result.size = int(size_str) if size_str else None + result.dist = self + return result + + @pass_none + def make_files(lines): + return list(starmap(make_file, csv.reader(lines))) + + return make_files(self._read_files_distinfo() or self._read_files_egginfo()) + + def _read_files_distinfo(self): + """ + Read the lines of RECORD + """ + text = self.read_text('RECORD') + return text and text.splitlines() + + def _read_files_egginfo(self): + """ + SOURCES.txt might contain literal commas, so wrap each line + in quotes. + """ + text = self.read_text('SOURCES.txt') + return text and map('"{}"'.format, text.splitlines()) + + @property + def requires(self): + """Generated requirements specified for this Distribution""" + reqs = self._read_dist_info_reqs() or self._read_egg_info_reqs() + return reqs and list(reqs) + + def _read_dist_info_reqs(self): + return self.metadata.get_all('Requires-Dist') + + def _read_egg_info_reqs(self): + source = self.read_text('requires.txt') + return source and self._deps_from_requires_text(source) + + @classmethod + def _deps_from_requires_text(cls, source): + return cls._convert_egg_info_reqs_to_simple_reqs(Sectioned.read(source)) + + @staticmethod + def _convert_egg_info_reqs_to_simple_reqs(sections): + """ + Historically, setuptools would solicit and store 'extra' + requirements, including those with environment markers, + in separate sections. More modern tools expect each + dependency to be defined separately, with any relevant + extras and environment markers attached directly to that + requirement. This method converts the former to the + latter. See _test_deps_from_requires_text for an example. + """ + + def make_condition(name): + return name and f'extra == "{name}"' + + def quoted_marker(section): + section = section or '' + extra, sep, markers = section.partition(':') + if extra and markers: + markers = f'({markers})' + conditions = list(filter(None, [markers, make_condition(extra)])) + return '; ' + ' and '.join(conditions) if conditions else '' + + def url_req_space(req): + """ + PEP 508 requires a space between the url_spec and the quoted_marker. + Ref python/importlib_metadata#357. + """ + # '@' is uniquely indicative of a url_req. + return ' ' * ('@' in req) + + for section in sections: + space = url_req_space(section.value) + yield section.value + space + quoted_marker(section.name) + + +class DistributionFinder(MetaPathFinder): + """ + A MetaPathFinder capable of discovering installed distributions. + """ + + class Context: + """ + Keyword arguments presented by the caller to + ``distributions()`` or ``Distribution.discover()`` + to narrow the scope of a search for distributions + in all DistributionFinders. + + Each DistributionFinder may expect any parameters + and should attempt to honor the canonical + parameters defined below when appropriate. + """ + + name = None + """ + Specific name for which a distribution finder should match. + A name of ``None`` matches all distributions. + """ + + def __init__(self, **kwargs): + vars(self).update(kwargs) + + @property + def path(self): + """ + The sequence of directory path that a distribution finder + should search. + + Typically refers to Python installed package paths such as + "site-packages" directories and defaults to ``sys.path``. + """ + return vars(self).get('path', sys.path) + + @abc.abstractmethod + def find_distributions(self, context=Context()): + """ + Find distributions. + + Return an iterable of all Distribution instances capable of + loading the metadata for packages matching the ``context``, + a DistributionFinder.Context instance. + """ + + +class FastPath: + """ + Micro-optimized class for searching a path for + children. + + >>> FastPath('').children() + ['...'] + """ + + @functools.lru_cache() # type: ignore + def __new__(cls, root): + return super().__new__(cls) + + def __init__(self, root): + self.root = str(root) + + def joinpath(self, child): + return pathlib.Path(self.root, child) + + def children(self): + with suppress(Exception): + return os.listdir(self.root or '.') + with suppress(Exception): + return self.zip_children() + return [] + + def zip_children(self): + zip_path = zipp.Path(self.root) + names = zip_path.root.namelist() + self.joinpath = zip_path.joinpath + + return dict.fromkeys(child.split(posixpath.sep, 1)[0] for child in names) + + def search(self, name): + return self.lookup(self.mtime).search(name) + + @property + def mtime(self): + with suppress(OSError): + return os.stat(self.root).st_mtime + self.lookup.cache_clear() + + @method_cache + def lookup(self, mtime): + return Lookup(self) + + +class Lookup: + def __init__(self, path: FastPath): + base = os.path.basename(path.root).lower() + base_is_egg = base.endswith(".egg") + self.infos = FreezableDefaultDict(list) + self.eggs = FreezableDefaultDict(list) + + for child in path.children(): + low = child.lower() + if low.endswith((".dist-info", ".egg-info")): + # rpartition is faster than splitext and suitable for this purpose. + name = low.rpartition(".")[0].partition("-")[0] + normalized = Prepared.normalize(name) + self.infos[normalized].append(path.joinpath(child)) + elif base_is_egg and low == "egg-info": + name = base.rpartition(".")[0].partition("-")[0] + legacy_normalized = Prepared.legacy_normalize(name) + self.eggs[legacy_normalized].append(path.joinpath(child)) + + self.infos.freeze() + self.eggs.freeze() + + def search(self, prepared): + infos = ( + self.infos[prepared.normalized] + if prepared + else itertools.chain.from_iterable(self.infos.values()) + ) + eggs = ( + self.eggs[prepared.legacy_normalized] + if prepared + else itertools.chain.from_iterable(self.eggs.values()) + ) + return itertools.chain(infos, eggs) + + +class Prepared: + """ + A prepared search for metadata on a possibly-named package. + """ + + normalized = None + legacy_normalized = None + + def __init__(self, name): + self.name = name + if name is None: + return + self.normalized = self.normalize(name) + self.legacy_normalized = self.legacy_normalize(name) + + @staticmethod + def normalize(name): + """ + PEP 503 normalization plus dashes as underscores. + """ + return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_') + + @staticmethod + def legacy_normalize(name): + """ + Normalize the package name as found in the convention in + older packaging tools versions and specs. + """ + return name.lower().replace('-', '_') + + def __bool__(self): + return bool(self.name) + + +@install +class MetadataPathFinder(NullFinder, DistributionFinder): + """A degenerate finder for distribution packages on the file system. + + This finder supplies only a find_distributions() method for versions + of Python that do not have a PathFinder find_distributions(). + """ + + def find_distributions(self, context=DistributionFinder.Context()): + """ + Find distributions. + + Return an iterable of all Distribution instances capable of + loading the metadata for packages matching ``context.name`` + (or all names if ``None`` indicated) along the paths in the list + of directories ``context.path``. + """ + found = self._search_paths(context.name, context.path) + return map(PathDistribution, found) + + @classmethod + def _search_paths(cls, name, paths): + """Find metadata directories in paths heuristically.""" + prepared = Prepared(name) + return itertools.chain.from_iterable( + path.search(prepared) for path in map(FastPath, paths) + ) + + def invalidate_caches(cls): + FastPath.__new__.cache_clear() + + +class PathDistribution(Distribution): + def __init__(self, path: SimplePath): + """Construct a distribution. + + :param path: SimplePath indicating the metadata directory. + """ + self._path = path + + def read_text(self, filename): + with suppress( + FileNotFoundError, + IsADirectoryError, + KeyError, + NotADirectoryError, + PermissionError, + ): + return self._path.joinpath(filename).read_text(encoding='utf-8') + + read_text.__doc__ = Distribution.read_text.__doc__ + + def locate_file(self, path): + return self._path.parent / path + + @property + def _normalized_name(self): + """ + Performance optimization: where possible, resolve the + normalized name from the file system path. + """ + stem = os.path.basename(str(self._path)) + return self._name_from_stem(stem) or super()._normalized_name + + def _name_from_stem(self, stem): + name, ext = os.path.splitext(stem) + if ext not in ('.dist-info', '.egg-info'): + return + name, sep, rest = stem.partition('-') + return name + + +def distribution(distribution_name): + """Get the ``Distribution`` instance for the named package. + + :param distribution_name: The name of the distribution package as a string. + :return: A ``Distribution`` instance (or subclass thereof). + """ + return Distribution.from_name(distribution_name) + + +def distributions(**kwargs): + """Get all ``Distribution`` instances in the current environment. + + :return: An iterable of ``Distribution`` instances. + """ + return Distribution.discover(**kwargs) + + +def metadata(distribution_name) -> _meta.PackageMetadata: + """Get the metadata for the named package. + + :param distribution_name: The name of the distribution package to query. + :return: A PackageMetadata containing the parsed metadata. + """ + return Distribution.from_name(distribution_name).metadata + + +def version(distribution_name): + """Get the version string for the named package. + + :param distribution_name: The name of the distribution package to query. + :return: The version string for the package as defined in the package's + "Version" metadata key. + """ + return distribution(distribution_name).version + + +def entry_points(**params) -> Union[EntryPoints, SelectableGroups]: + """Return EntryPoint objects for all installed packages. + + Pass selection parameters (group or name) to filter the + result to entry points matching those properties (see + EntryPoints.select()). + + For compatibility, returns ``SelectableGroups`` object unless + selection parameters are supplied. In the future, this function + will return ``EntryPoints`` instead of ``SelectableGroups`` + even when no selection parameters are supplied. + + For maximum future compatibility, pass selection parameters + or invoke ``.select`` with parameters on the result. + + :return: EntryPoints or SelectableGroups for all installed packages. + """ + norm_name = operator.attrgetter('_normalized_name') + unique = functools.partial(unique_everseen, key=norm_name) + eps = itertools.chain.from_iterable( + dist.entry_points for dist in unique(distributions()) + ) + return SelectableGroups.load(eps).select(**params) + + +def files(distribution_name): + """Return a list of files for the named package. + + :param distribution_name: The name of the distribution package to query. + :return: List of files composing the distribution. + """ + return distribution(distribution_name).files + + +def requires(distribution_name): + """ + Return a list of requirements for the named package. + + :return: An iterator of requirements, suitable for + packaging.requirement.Requirement. + """ + return distribution(distribution_name).requires + + +def packages_distributions() -> Mapping[str, List[str]]: + """ + Return a mapping of top-level packages to their + distributions. + + >>> import collections.abc + >>> pkgs = packages_distributions() + >>> all(isinstance(dist, collections.abc.Sequence) for dist in pkgs.values()) + True + """ + pkg_to_dist = collections.defaultdict(list) + for dist in distributions(): + for pkg in _top_level_declared(dist) or _top_level_inferred(dist): + pkg_to_dist[pkg].append(dist.metadata['Name']) + return dict(pkg_to_dist) + + +def _top_level_declared(dist): + return (dist.read_text('top_level.txt') or '').split() + + +def _top_level_inferred(dist): + return { + f.parts[0] if len(f.parts) > 1 else f.with_suffix('').name + for f in always_iterable(dist.files) + if f.suffix == ".py" + } diff --git a/metaflow/_vendor/v3_6/importlib_metadata/_adapters.py b/metaflow/_vendor/v3_6/importlib_metadata/_adapters.py new file mode 100644 index 00000000000..aa460d3eda5 --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/_adapters.py @@ -0,0 +1,68 @@ +import re +import textwrap +import email.message + +from ._text import FoldedCase + + +class Message(email.message.Message): + multiple_use_keys = set( + map( + FoldedCase, + [ + 'Classifier', + 'Obsoletes-Dist', + 'Platform', + 'Project-URL', + 'Provides-Dist', + 'Provides-Extra', + 'Requires-Dist', + 'Requires-External', + 'Supported-Platform', + 'Dynamic', + ], + ) + ) + """ + Keys that may be indicated multiple times per PEP 566. + """ + + def __new__(cls, orig: email.message.Message): + res = super().__new__(cls) + vars(res).update(vars(orig)) + return res + + def __init__(self, *args, **kwargs): + self._headers = self._repair_headers() + + # suppress spurious error from mypy + def __iter__(self): + return super().__iter__() + + def _repair_headers(self): + def redent(value): + "Correct for RFC822 indentation" + if not value or '\n' not in value: + return value + return textwrap.dedent(' ' * 8 + value) + + headers = [(key, redent(value)) for key, value in vars(self)['_headers']] + if self._payload: + headers.append(('Description', self.get_payload())) + return headers + + @property + def json(self): + """ + Convert PackageMetadata to a JSON-compatible format + per PEP 0566. + """ + + def transform(key): + value = self.get_all(key) if key in self.multiple_use_keys else self[key] + if key == 'Keywords': + value = re.split(r'\s+', value) + tk = key.lower().replace('-', '_') + return tk, value + + return dict(map(transform, map(FoldedCase, self))) diff --git a/metaflow/_vendor/v3_6/importlib_metadata/_collections.py b/metaflow/_vendor/v3_6/importlib_metadata/_collections.py new file mode 100644 index 00000000000..cf0954e1a30 --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/_collections.py @@ -0,0 +1,30 @@ +import collections + + +# from jaraco.collections 3.3 +class FreezableDefaultDict(collections.defaultdict): + """ + Often it is desirable to prevent the mutation of + a default dict after its initial construction, such + as to prevent mutation during iteration. + + >>> dd = FreezableDefaultDict(list) + >>> dd[0].append('1') + >>> dd.freeze() + >>> dd[1] + [] + >>> len(dd) + 1 + """ + + def __missing__(self, key): + return getattr(self, '_frozen', super().__missing__)(key) + + def freeze(self): + self._frozen = lambda key: self.default_factory() + + +class Pair(collections.namedtuple('Pair', 'name value')): + @classmethod + def parse(cls, text): + return cls(*map(str.strip, text.split("=", 1))) diff --git a/metaflow/_vendor/v3_6/importlib_metadata/_compat.py b/metaflow/_vendor/v3_6/importlib_metadata/_compat.py new file mode 100644 index 00000000000..3680940f0b0 --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/_compat.py @@ -0,0 +1,71 @@ +import sys +import platform + + +__all__ = ['install', 'NullFinder', 'Protocol'] + + +try: + from typing import Protocol +except ImportError: # pragma: no cover + from metaflow._vendor.v3_6.typing_extensions import Protocol # type: ignore + + +def install(cls): + """ + Class decorator for installation on sys.meta_path. + + Adds the backport DistributionFinder to sys.meta_path and + attempts to disable the finder functionality of the stdlib + DistributionFinder. + """ + sys.meta_path.append(cls()) + disable_stdlib_finder() + return cls + + +def disable_stdlib_finder(): + """ + Give the backport primacy for discovering path-based distributions + by monkey-patching the stdlib O_O. + + See #91 for more background for rationale on this sketchy + behavior. + """ + + def matches(finder): + return getattr( + finder, '__module__', None + ) == '_frozen_importlib_external' and hasattr(finder, 'find_distributions') + + for finder in filter(matches, sys.meta_path): # pragma: nocover + del finder.find_distributions + + +class NullFinder: + """ + A "Finder" (aka "MetaClassFinder") that never finds any modules, + but may find distributions. + """ + + @staticmethod + def find_spec(*args, **kwargs): + return None + + # In Python 2, the import system requires finders + # to have a find_module() method, but this usage + # is deprecated in Python 3 in favor of find_spec(). + # For the purposes of this finder (i.e. being present + # on sys.meta_path but having no other import + # system functionality), the two methods are identical. + find_module = find_spec + + +def pypy_partial(val): + """ + Adjust for variable stacklevel on partial under PyPy. + + Workaround for #327. + """ + is_pypy = platform.python_implementation() == 'PyPy' + return val + is_pypy diff --git a/metaflow/_vendor/v3_6/importlib_metadata/_functools.py b/metaflow/_vendor/v3_6/importlib_metadata/_functools.py new file mode 100644 index 00000000000..71f66bd03cb --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/_functools.py @@ -0,0 +1,104 @@ +import types +import functools + + +# from jaraco.functools 3.3 +def method_cache(method, cache_wrapper=None): + """ + Wrap lru_cache to support storing the cache data in the object instances. + + Abstracts the common paradigm where the method explicitly saves an + underscore-prefixed protected property on first call and returns that + subsequently. + + >>> class MyClass: + ... calls = 0 + ... + ... @method_cache + ... def method(self, value): + ... self.calls += 1 + ... return value + + >>> a = MyClass() + >>> a.method(3) + 3 + >>> for x in range(75): + ... res = a.method(x) + >>> a.calls + 75 + + Note that the apparent behavior will be exactly like that of lru_cache + except that the cache is stored on each instance, so values in one + instance will not flush values from another, and when an instance is + deleted, so are the cached values for that instance. + + >>> b = MyClass() + >>> for x in range(35): + ... res = b.method(x) + >>> b.calls + 35 + >>> a.method(0) + 0 + >>> a.calls + 75 + + Note that if method had been decorated with ``functools.lru_cache()``, + a.calls would have been 76 (due to the cached value of 0 having been + flushed by the 'b' instance). + + Clear the cache with ``.cache_clear()`` + + >>> a.method.cache_clear() + + Same for a method that hasn't yet been called. + + >>> c = MyClass() + >>> c.method.cache_clear() + + Another cache wrapper may be supplied: + + >>> cache = functools.lru_cache(maxsize=2) + >>> MyClass.method2 = method_cache(lambda self: 3, cache_wrapper=cache) + >>> a = MyClass() + >>> a.method2() + 3 + + Caution - do not subsequently wrap the method with another decorator, such + as ``@property``, which changes the semantics of the function. + + See also + http://code.activestate.com/recipes/577452-a-memoize-decorator-for-instance-methods/ + for another implementation and additional justification. + """ + cache_wrapper = cache_wrapper or functools.lru_cache() + + def wrapper(self, *args, **kwargs): + # it's the first call, replace the method with a cached, bound method + bound_method = types.MethodType(method, self) + cached_method = cache_wrapper(bound_method) + setattr(self, method.__name__, cached_method) + return cached_method(*args, **kwargs) + + # Support cache clear even before cache has been created. + wrapper.cache_clear = lambda: None + + return wrapper + + +# From jaraco.functools 3.3 +def pass_none(func): + """ + Wrap func so it's not called if its first param is None + + >>> print_text = pass_none(print) + >>> print_text('text') + text + >>> print_text(None) + """ + + @functools.wraps(func) + def wrapper(param, *args, **kwargs): + if param is not None: + return func(param, *args, **kwargs) + + return wrapper diff --git a/metaflow/_vendor/v3_6/importlib_metadata/_itertools.py b/metaflow/_vendor/v3_6/importlib_metadata/_itertools.py new file mode 100644 index 00000000000..d4ca9b9140e --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/_itertools.py @@ -0,0 +1,73 @@ +from itertools import filterfalse + + +def unique_everseen(iterable, key=None): + "List unique elements, preserving order. Remember all elements ever seen." + # unique_everseen('AAAABBBCCDAABBB') --> A B C D + # unique_everseen('ABBCcAD', str.lower) --> A B C D + seen = set() + seen_add = seen.add + if key is None: + for element in filterfalse(seen.__contains__, iterable): + seen_add(element) + yield element + else: + for element in iterable: + k = key(element) + if k not in seen: + seen_add(k) + yield element + + +# copied from more_itertools 8.8 +def always_iterable(obj, base_type=(str, bytes)): + """If *obj* is iterable, return an iterator over its items:: + + >>> obj = (1, 2, 3) + >>> list(always_iterable(obj)) + [1, 2, 3] + + If *obj* is not iterable, return a one-item iterable containing *obj*:: + + >>> obj = 1 + >>> list(always_iterable(obj)) + [1] + + If *obj* is ``None``, return an empty iterable: + + >>> obj = None + >>> list(always_iterable(None)) + [] + + By default, binary and text strings are not considered iterable:: + + >>> obj = 'foo' + >>> list(always_iterable(obj)) + ['foo'] + + If *base_type* is set, objects for which ``isinstance(obj, base_type)`` + returns ``True`` won't be considered iterable. + + >>> obj = {'a': 1} + >>> list(always_iterable(obj)) # Iterate over the dict's keys + ['a'] + >>> list(always_iterable(obj, base_type=dict)) # Treat dicts as a unit + [{'a': 1}] + + Set *base_type* to ``None`` to avoid any special handling and treat objects + Python considers iterable as iterable: + + >>> obj = 'foo' + >>> list(always_iterable(obj, base_type=None)) + ['f', 'o', 'o'] + """ + if obj is None: + return iter(()) + + if (base_type is not None) and isinstance(obj, base_type): + return iter((obj,)) + + try: + return iter(obj) + except TypeError: + return iter((obj,)) diff --git a/metaflow/_vendor/v3_6/importlib_metadata/_meta.py b/metaflow/_vendor/v3_6/importlib_metadata/_meta.py new file mode 100644 index 00000000000..37ee43e6ef4 --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/_meta.py @@ -0,0 +1,48 @@ +from ._compat import Protocol +from typing import Any, Dict, Iterator, List, TypeVar, Union + + +_T = TypeVar("_T") + + +class PackageMetadata(Protocol): + def __len__(self) -> int: + ... # pragma: no cover + + def __contains__(self, item: str) -> bool: + ... # pragma: no cover + + def __getitem__(self, key: str) -> str: + ... # pragma: no cover + + def __iter__(self) -> Iterator[str]: + ... # pragma: no cover + + def get_all(self, name: str, failobj: _T = ...) -> Union[List[Any], _T]: + """ + Return all values associated with a possibly multi-valued key. + """ + + @property + def json(self) -> Dict[str, Union[str, List[str]]]: + """ + A JSON-compatible form of the metadata. + """ + + +class SimplePath(Protocol): + """ + A minimal subset of pathlib.Path required by PathDistribution. + """ + + def joinpath(self) -> 'SimplePath': + ... # pragma: no cover + + def __truediv__(self) -> 'SimplePath': + ... # pragma: no cover + + def parent(self) -> 'SimplePath': + ... # pragma: no cover + + def read_text(self) -> str: + ... # pragma: no cover diff --git a/metaflow/_vendor/v3_6/importlib_metadata/_text.py b/metaflow/_vendor/v3_6/importlib_metadata/_text.py new file mode 100644 index 00000000000..c88cfbb2349 --- /dev/null +++ b/metaflow/_vendor/v3_6/importlib_metadata/_text.py @@ -0,0 +1,99 @@ +import re + +from ._functools import method_cache + + +# from jaraco.text 3.5 +class FoldedCase(str): + """ + A case insensitive string class; behaves just like str + except compares equal when the only variation is case. + + >>> s = FoldedCase('hello world') + + >>> s == 'Hello World' + True + + >>> 'Hello World' == s + True + + >>> s != 'Hello World' + False + + >>> s.index('O') + 4 + + >>> s.split('O') + ['hell', ' w', 'rld'] + + >>> sorted(map(FoldedCase, ['GAMMA', 'alpha', 'Beta'])) + ['alpha', 'Beta', 'GAMMA'] + + Sequence membership is straightforward. + + >>> "Hello World" in [s] + True + >>> s in ["Hello World"] + True + + You may test for set inclusion, but candidate and elements + must both be folded. + + >>> FoldedCase("Hello World") in {s} + True + >>> s in {FoldedCase("Hello World")} + True + + String inclusion works as long as the FoldedCase object + is on the right. + + >>> "hello" in FoldedCase("Hello World") + True + + But not if the FoldedCase object is on the left: + + >>> FoldedCase('hello') in 'Hello World' + False + + In that case, use in_: + + >>> FoldedCase('hello').in_('Hello World') + True + + >>> FoldedCase('hello') > FoldedCase('Hello') + False + """ + + def __lt__(self, other): + return self.lower() < other.lower() + + def __gt__(self, other): + return self.lower() > other.lower() + + def __eq__(self, other): + return self.lower() == other.lower() + + def __ne__(self, other): + return self.lower() != other.lower() + + def __hash__(self): + return hash(self.lower()) + + def __contains__(self, other): + return super().lower().__contains__(other.lower()) + + def in_(self, other): + "Does self appear in other?" + return self in FoldedCase(other) + + # cache lower since it's likely to be called frequently. + @method_cache + def lower(self): + return super().lower() + + def index(self, sub): + return self.lower().index(sub.lower()) + + def split(self, splitter=' ', maxsplit=0): + pattern = re.compile(re.escape(splitter), re.I) + return pattern.split(self, maxsplit) diff --git a/metaflow/plugins/aws/eks/__init__.py b/metaflow/_vendor/v3_6/importlib_metadata/py.typed similarity index 100% rename from metaflow/plugins/aws/eks/__init__.py rename to metaflow/_vendor/v3_6/importlib_metadata/py.typed diff --git a/metaflow/_vendor/v3_6/typing_extensions.LICENSE b/metaflow/_vendor/v3_6/typing_extensions.LICENSE new file mode 100644 index 00000000000..583f9f6e617 --- /dev/null +++ b/metaflow/_vendor/v3_6/typing_extensions.LICENSE @@ -0,0 +1,254 @@ +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations (now Zope +Corporation, see http://www.zope.com). In 2001, the Python Software +Foundation (PSF, see http://www.python.org/psf/) was formed, a +non-profit organization created specifically to own Python-related +Intellectual Property. Zope Corporation is a sponsoring member of +the PSF. + +All Python releases are Open Source (see http://www.opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2 and above 2.1.1 2001-now PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are +retained in Python alone or in any derivative version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the Internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the Internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/metaflow/_vendor/v3_6/typing_extensions.py b/metaflow/_vendor/v3_6/typing_extensions.py new file mode 100644 index 00000000000..43c05bdcd22 --- /dev/null +++ b/metaflow/_vendor/v3_6/typing_extensions.py @@ -0,0 +1,2908 @@ +import abc +import collections +import collections.abc +import operator +import sys +import types as _types +import typing + +# After PEP 560, internal typing API was substantially reworked. +# This is especially important for Protocol class which uses internal APIs +# quite extensively. +PEP_560 = sys.version_info[:3] >= (3, 7, 0) + +if PEP_560: + GenericMeta = type +else: + # 3.6 + from typing import GenericMeta, _type_vars # noqa + + +# Please keep __all__ alphabetized within each category. +__all__ = [ + # Super-special typing primitives. + 'ClassVar', + 'Concatenate', + 'Final', + 'LiteralString', + 'ParamSpec', + 'Self', + 'Type', + 'TypeVarTuple', + 'Unpack', + + # ABCs (from collections.abc). + 'Awaitable', + 'AsyncIterator', + 'AsyncIterable', + 'Coroutine', + 'AsyncGenerator', + 'AsyncContextManager', + 'ChainMap', + + # Concrete collection types. + 'ContextManager', + 'Counter', + 'Deque', + 'DefaultDict', + 'OrderedDict', + 'TypedDict', + + # Structural checks, a.k.a. protocols. + 'SupportsIndex', + + # One-off things. + 'Annotated', + 'assert_never', + 'dataclass_transform', + 'final', + 'IntVar', + 'is_typeddict', + 'Literal', + 'NewType', + 'overload', + 'Protocol', + 'reveal_type', + 'runtime', + 'runtime_checkable', + 'Text', + 'TypeAlias', + 'TypeGuard', + 'TYPE_CHECKING', + 'Never', + 'NoReturn', + 'Required', + 'NotRequired', +] + +if PEP_560: + __all__.extend(["get_args", "get_origin", "get_type_hints"]) + +# The functions below are modified copies of typing internal helpers. +# They are needed by _ProtocolMeta and they provide support for PEP 646. + + +def _no_slots_copy(dct): + dict_copy = dict(dct) + if '__slots__' in dict_copy: + for slot in dict_copy['__slots__']: + dict_copy.pop(slot, None) + return dict_copy + + +_marker = object() + + +def _check_generic(cls, parameters, elen=_marker): + """Check correct count for parameters of a generic cls (internal helper). + This gives a nice error message in case of count mismatch. + """ + if not elen: + raise TypeError(f"{cls} is not a generic class") + if elen is _marker: + if not hasattr(cls, "__parameters__") or not cls.__parameters__: + raise TypeError(f"{cls} is not a generic class") + elen = len(cls.__parameters__) + alen = len(parameters) + if alen != elen: + if hasattr(cls, "__parameters__"): + parameters = [p for p in cls.__parameters__ if not _is_unpack(p)] + num_tv_tuples = sum(isinstance(p, TypeVarTuple) for p in parameters) + if (num_tv_tuples > 0) and (alen >= elen - num_tv_tuples): + return + raise TypeError(f"Too {'many' if alen > elen else 'few'} parameters for {cls};" + f" actual {alen}, expected {elen}") + + +if sys.version_info >= (3, 10): + def _should_collect_from_parameters(t): + return isinstance( + t, (typing._GenericAlias, _types.GenericAlias, _types.UnionType) + ) +elif sys.version_info >= (3, 9): + def _should_collect_from_parameters(t): + return isinstance(t, (typing._GenericAlias, _types.GenericAlias)) +else: + def _should_collect_from_parameters(t): + return isinstance(t, typing._GenericAlias) and not t._special + + +def _collect_type_vars(types, typevar_types=None): + """Collect all type variable contained in types in order of + first appearance (lexicographic order). For example:: + + _collect_type_vars((T, List[S, T])) == (T, S) + """ + if typevar_types is None: + typevar_types = typing.TypeVar + tvars = [] + for t in types: + if ( + isinstance(t, typevar_types) and + t not in tvars and + not _is_unpack(t) + ): + tvars.append(t) + if _should_collect_from_parameters(t): + tvars.extend([t for t in t.__parameters__ if t not in tvars]) + return tuple(tvars) + + +# 3.6.2+ +if hasattr(typing, 'NoReturn'): + NoReturn = typing.NoReturn +# 3.6.0-3.6.1 +else: + class _NoReturn(typing._FinalTypingBase, _root=True): + """Special type indicating functions that never return. + Example:: + + from typing import NoReturn + + def stop() -> NoReturn: + raise Exception('no way') + + This type is invalid in other positions, e.g., ``List[NoReturn]`` + will fail in static type checkers. + """ + __slots__ = () + + def __instancecheck__(self, obj): + raise TypeError("NoReturn cannot be used with isinstance().") + + def __subclasscheck__(self, cls): + raise TypeError("NoReturn cannot be used with issubclass().") + + NoReturn = _NoReturn(_root=True) + +# Some unconstrained type variables. These are used by the container types. +# (These are not for export.) +T = typing.TypeVar('T') # Any type. +KT = typing.TypeVar('KT') # Key type. +VT = typing.TypeVar('VT') # Value type. +T_co = typing.TypeVar('T_co', covariant=True) # Any type covariant containers. +T_contra = typing.TypeVar('T_contra', contravariant=True) # Ditto contravariant. + +ClassVar = typing.ClassVar + +# On older versions of typing there is an internal class named "Final". +# 3.8+ +if hasattr(typing, 'Final') and sys.version_info[:2] >= (3, 7): + Final = typing.Final +# 3.7 +elif sys.version_info[:2] >= (3, 7): + class _FinalForm(typing._SpecialForm, _root=True): + + def __repr__(self): + return 'typing_extensions.' + self._name + + def __getitem__(self, parameters): + item = typing._type_check(parameters, + f'{self._name} accepts only single type') + return typing._GenericAlias(self, (item,)) + + Final = _FinalForm('Final', + doc="""A special typing construct to indicate that a name + cannot be re-assigned or overridden in a subclass. + For example: + + MAX_SIZE: Final = 9000 + MAX_SIZE += 1 # Error reported by type checker + + class Connection: + TIMEOUT: Final[int] = 10 + class FastConnector(Connection): + TIMEOUT = 1 # Error reported by type checker + + There is no runtime checking of these properties.""") +# 3.6 +else: + class _Final(typing._FinalTypingBase, _root=True): + """A special typing construct to indicate that a name + cannot be re-assigned or overridden in a subclass. + For example: + + MAX_SIZE: Final = 9000 + MAX_SIZE += 1 # Error reported by type checker + + class Connection: + TIMEOUT: Final[int] = 10 + class FastConnector(Connection): + TIMEOUT = 1 # Error reported by type checker + + There is no runtime checking of these properties. + """ + + __slots__ = ('__type__',) + + def __init__(self, tp=None, **kwds): + self.__type__ = tp + + def __getitem__(self, item): + cls = type(self) + if self.__type__ is None: + return cls(typing._type_check(item, + f'{cls.__name__[1:]} accepts only single type.'), + _root=True) + raise TypeError(f'{cls.__name__[1:]} cannot be further subscripted') + + def _eval_type(self, globalns, localns): + new_tp = typing._eval_type(self.__type__, globalns, localns) + if new_tp == self.__type__: + return self + return type(self)(new_tp, _root=True) + + def __repr__(self): + r = super().__repr__() + if self.__type__ is not None: + r += f'[{typing._type_repr(self.__type__)}]' + return r + + def __hash__(self): + return hash((type(self).__name__, self.__type__)) + + def __eq__(self, other): + if not isinstance(other, _Final): + return NotImplemented + if self.__type__ is not None: + return self.__type__ == other.__type__ + return self is other + + Final = _Final(_root=True) + + +if sys.version_info >= (3, 11): + final = typing.final +else: + # @final exists in 3.8+, but we backport it for all versions + # before 3.11 to keep support for the __final__ attribute. + # See https://bugs.python.org/issue46342 + def final(f): + """This decorator can be used to indicate to type checkers that + the decorated method cannot be overridden, and decorated class + cannot be subclassed. For example: + + class Base: + @final + def done(self) -> None: + ... + class Sub(Base): + def done(self) -> None: # Error reported by type checker + ... + @final + class Leaf: + ... + class Other(Leaf): # Error reported by type checker + ... + + There is no runtime checking of these properties. The decorator + sets the ``__final__`` attribute to ``True`` on the decorated object + to allow runtime introspection. + """ + try: + f.__final__ = True + except (AttributeError, TypeError): + # Skip the attribute silently if it is not writable. + # AttributeError happens if the object has __slots__ or a + # read-only property, TypeError if it's a builtin class. + pass + return f + + +def IntVar(name): + return typing.TypeVar(name) + + +# 3.8+: +if hasattr(typing, 'Literal'): + Literal = typing.Literal +# 3.7: +elif sys.version_info[:2] >= (3, 7): + class _LiteralForm(typing._SpecialForm, _root=True): + + def __repr__(self): + return 'typing_extensions.' + self._name + + def __getitem__(self, parameters): + return typing._GenericAlias(self, parameters) + + Literal = _LiteralForm('Literal', + doc="""A type that can be used to indicate to type checkers + that the corresponding value has a value literally equivalent + to the provided parameter. For example: + + var: Literal[4] = 4 + + The type checker understands that 'var' is literally equal to + the value 4 and no other value. + + Literal[...] cannot be subclassed. There is no runtime + checking verifying that the parameter is actually a value + instead of a type.""") +# 3.6: +else: + class _Literal(typing._FinalTypingBase, _root=True): + """A type that can be used to indicate to type checkers that the + corresponding value has a value literally equivalent to the + provided parameter. For example: + + var: Literal[4] = 4 + + The type checker understands that 'var' is literally equal to the + value 4 and no other value. + + Literal[...] cannot be subclassed. There is no runtime checking + verifying that the parameter is actually a value instead of a type. + """ + + __slots__ = ('__values__',) + + def __init__(self, values=None, **kwds): + self.__values__ = values + + def __getitem__(self, values): + cls = type(self) + if self.__values__ is None: + if not isinstance(values, tuple): + values = (values,) + return cls(values, _root=True) + raise TypeError(f'{cls.__name__[1:]} cannot be further subscripted') + + def _eval_type(self, globalns, localns): + return self + + def __repr__(self): + r = super().__repr__() + if self.__values__ is not None: + r += f'[{", ".join(map(typing._type_repr, self.__values__))}]' + return r + + def __hash__(self): + return hash((type(self).__name__, self.__values__)) + + def __eq__(self, other): + if not isinstance(other, _Literal): + return NotImplemented + if self.__values__ is not None: + return self.__values__ == other.__values__ + return self is other + + Literal = _Literal(_root=True) + + +_overload_dummy = typing._overload_dummy # noqa +overload = typing.overload + + +# This is not a real generic class. Don't use outside annotations. +Type = typing.Type + +# Various ABCs mimicking those in collections.abc. +# A few are simply re-exported for completeness. + + +class _ExtensionsGenericMeta(GenericMeta): + def __subclasscheck__(self, subclass): + """This mimics a more modern GenericMeta.__subclasscheck__() logic + (that does not have problems with recursion) to work around interactions + between collections, typing, and typing_extensions on older + versions of Python, see https://github.com/python/typing/issues/501. + """ + if self.__origin__ is not None: + if sys._getframe(1).f_globals['__name__'] not in ['abc', 'functools']: + raise TypeError("Parameterized generics cannot be used with class " + "or instance checks") + return False + if not self.__extra__: + return super().__subclasscheck__(subclass) + res = self.__extra__.__subclasshook__(subclass) + if res is not NotImplemented: + return res + if self.__extra__ in subclass.__mro__: + return True + for scls in self.__extra__.__subclasses__(): + if isinstance(scls, GenericMeta): + continue + if issubclass(subclass, scls): + return True + return False + + +Awaitable = typing.Awaitable +Coroutine = typing.Coroutine +AsyncIterable = typing.AsyncIterable +AsyncIterator = typing.AsyncIterator + +# 3.6.1+ +if hasattr(typing, 'Deque'): + Deque = typing.Deque +# 3.6.0 +else: + class Deque(collections.deque, typing.MutableSequence[T], + metaclass=_ExtensionsGenericMeta, + extra=collections.deque): + __slots__ = () + + def __new__(cls, *args, **kwds): + if cls._gorg is Deque: + return collections.deque(*args, **kwds) + return typing._generic_new(collections.deque, cls, *args, **kwds) + +ContextManager = typing.ContextManager +# 3.6.2+ +if hasattr(typing, 'AsyncContextManager'): + AsyncContextManager = typing.AsyncContextManager +# 3.6.0-3.6.1 +else: + from _collections_abc import _check_methods as _check_methods_in_mro # noqa + + class AsyncContextManager(typing.Generic[T_co]): + __slots__ = () + + async def __aenter__(self): + return self + + @abc.abstractmethod + async def __aexit__(self, exc_type, exc_value, traceback): + return None + + @classmethod + def __subclasshook__(cls, C): + if cls is AsyncContextManager: + return _check_methods_in_mro(C, "__aenter__", "__aexit__") + return NotImplemented + +DefaultDict = typing.DefaultDict + +# 3.7.2+ +if hasattr(typing, 'OrderedDict'): + OrderedDict = typing.OrderedDict +# 3.7.0-3.7.2 +elif (3, 7, 0) <= sys.version_info[:3] < (3, 7, 2): + OrderedDict = typing._alias(collections.OrderedDict, (KT, VT)) +# 3.6 +else: + class OrderedDict(collections.OrderedDict, typing.MutableMapping[KT, VT], + metaclass=_ExtensionsGenericMeta, + extra=collections.OrderedDict): + + __slots__ = () + + def __new__(cls, *args, **kwds): + if cls._gorg is OrderedDict: + return collections.OrderedDict(*args, **kwds) + return typing._generic_new(collections.OrderedDict, cls, *args, **kwds) + +# 3.6.2+ +if hasattr(typing, 'Counter'): + Counter = typing.Counter +# 3.6.0-3.6.1 +else: + class Counter(collections.Counter, + typing.Dict[T, int], + metaclass=_ExtensionsGenericMeta, extra=collections.Counter): + + __slots__ = () + + def __new__(cls, *args, **kwds): + if cls._gorg is Counter: + return collections.Counter(*args, **kwds) + return typing._generic_new(collections.Counter, cls, *args, **kwds) + +# 3.6.1+ +if hasattr(typing, 'ChainMap'): + ChainMap = typing.ChainMap +elif hasattr(collections, 'ChainMap'): + class ChainMap(collections.ChainMap, typing.MutableMapping[KT, VT], + metaclass=_ExtensionsGenericMeta, + extra=collections.ChainMap): + + __slots__ = () + + def __new__(cls, *args, **kwds): + if cls._gorg is ChainMap: + return collections.ChainMap(*args, **kwds) + return typing._generic_new(collections.ChainMap, cls, *args, **kwds) + +# 3.6.1+ +if hasattr(typing, 'AsyncGenerator'): + AsyncGenerator = typing.AsyncGenerator +# 3.6.0 +else: + class AsyncGenerator(AsyncIterator[T_co], typing.Generic[T_co, T_contra], + metaclass=_ExtensionsGenericMeta, + extra=collections.abc.AsyncGenerator): + __slots__ = () + +NewType = typing.NewType +Text = typing.Text +TYPE_CHECKING = typing.TYPE_CHECKING + + +def _gorg(cls): + """This function exists for compatibility with old typing versions.""" + assert isinstance(cls, GenericMeta) + if hasattr(cls, '_gorg'): + return cls._gorg + while cls.__origin__ is not None: + cls = cls.__origin__ + return cls + + +_PROTO_WHITELIST = ['Callable', 'Awaitable', + 'Iterable', 'Iterator', 'AsyncIterable', 'AsyncIterator', + 'Hashable', 'Sized', 'Container', 'Collection', 'Reversible', + 'ContextManager', 'AsyncContextManager'] + + +def _get_protocol_attrs(cls): + attrs = set() + for base in cls.__mro__[:-1]: # without object + if base.__name__ in ('Protocol', 'Generic'): + continue + annotations = getattr(base, '__annotations__', {}) + for attr in list(base.__dict__.keys()) + list(annotations.keys()): + if (not attr.startswith('_abc_') and attr not in ( + '__abstractmethods__', '__annotations__', '__weakref__', + '_is_protocol', '_is_runtime_protocol', '__dict__', + '__args__', '__slots__', + '__next_in_mro__', '__parameters__', '__origin__', + '__orig_bases__', '__extra__', '__tree_hash__', + '__doc__', '__subclasshook__', '__init__', '__new__', + '__module__', '_MutableMapping__marker', '_gorg')): + attrs.add(attr) + return attrs + + +def _is_callable_members_only(cls): + return all(callable(getattr(cls, attr, None)) for attr in _get_protocol_attrs(cls)) + + +# 3.8+ +if hasattr(typing, 'Protocol'): + Protocol = typing.Protocol +# 3.7 +elif PEP_560: + + def _no_init(self, *args, **kwargs): + if type(self)._is_protocol: + raise TypeError('Protocols cannot be instantiated') + + class _ProtocolMeta(abc.ABCMeta): + # This metaclass is a bit unfortunate and exists only because of the lack + # of __instancehook__. + def __instancecheck__(cls, instance): + # We need this method for situations where attributes are + # assigned in __init__. + if ((not getattr(cls, '_is_protocol', False) or + _is_callable_members_only(cls)) and + issubclass(instance.__class__, cls)): + return True + if cls._is_protocol: + if all(hasattr(instance, attr) and + (not callable(getattr(cls, attr, None)) or + getattr(instance, attr) is not None) + for attr in _get_protocol_attrs(cls)): + return True + return super().__instancecheck__(instance) + + class Protocol(metaclass=_ProtocolMeta): + # There is quite a lot of overlapping code with typing.Generic. + # Unfortunately it is hard to avoid this while these live in two different + # modules. The duplicated code will be removed when Protocol is moved to typing. + """Base class for protocol classes. Protocol classes are defined as:: + + class Proto(Protocol): + def meth(self) -> int: + ... + + Such classes are primarily used with static type checkers that recognize + structural subtyping (static duck-typing), for example:: + + class C: + def meth(self) -> int: + return 0 + + def func(x: Proto) -> int: + return x.meth() + + func(C()) # Passes static type check + + See PEP 544 for details. Protocol classes decorated with + @typing_extensions.runtime act as simple-minded runtime protocol that checks + only the presence of given attributes, ignoring their type signatures. + + Protocol classes can be generic, they are defined as:: + + class GenProto(Protocol[T]): + def meth(self) -> T: + ... + """ + __slots__ = () + _is_protocol = True + + def __new__(cls, *args, **kwds): + if cls is Protocol: + raise TypeError("Type Protocol cannot be instantiated; " + "it can only be used as a base class") + return super().__new__(cls) + + @typing._tp_cache + def __class_getitem__(cls, params): + if not isinstance(params, tuple): + params = (params,) + if not params and cls is not typing.Tuple: + raise TypeError( + f"Parameter list to {cls.__qualname__}[...] cannot be empty") + msg = "Parameters to generic types must be types." + params = tuple(typing._type_check(p, msg) for p in params) # noqa + if cls is Protocol: + # Generic can only be subscripted with unique type variables. + if not all(isinstance(p, typing.TypeVar) for p in params): + i = 0 + while isinstance(params[i], typing.TypeVar): + i += 1 + raise TypeError( + "Parameters to Protocol[...] must all be type variables." + f" Parameter {i + 1} is {params[i]}") + if len(set(params)) != len(params): + raise TypeError( + "Parameters to Protocol[...] must all be unique") + else: + # Subscripting a regular Generic subclass. + _check_generic(cls, params, len(cls.__parameters__)) + return typing._GenericAlias(cls, params) + + def __init_subclass__(cls, *args, **kwargs): + tvars = [] + if '__orig_bases__' in cls.__dict__: + error = typing.Generic in cls.__orig_bases__ + else: + error = typing.Generic in cls.__bases__ + if error: + raise TypeError("Cannot inherit from plain Generic") + if '__orig_bases__' in cls.__dict__: + tvars = typing._collect_type_vars(cls.__orig_bases__) + # Look for Generic[T1, ..., Tn] or Protocol[T1, ..., Tn]. + # If found, tvars must be a subset of it. + # If not found, tvars is it. + # Also check for and reject plain Generic, + # and reject multiple Generic[...] and/or Protocol[...]. + gvars = None + for base in cls.__orig_bases__: + if (isinstance(base, typing._GenericAlias) and + base.__origin__ in (typing.Generic, Protocol)): + # for error messages + the_base = base.__origin__.__name__ + if gvars is not None: + raise TypeError( + "Cannot inherit from Generic[...]" + " and/or Protocol[...] multiple types.") + gvars = base.__parameters__ + if gvars is None: + gvars = tvars + else: + tvarset = set(tvars) + gvarset = set(gvars) + if not tvarset <= gvarset: + s_vars = ', '.join(str(t) for t in tvars if t not in gvarset) + s_args = ', '.join(str(g) for g in gvars) + raise TypeError(f"Some type variables ({s_vars}) are" + f" not listed in {the_base}[{s_args}]") + tvars = gvars + cls.__parameters__ = tuple(tvars) + + # Determine if this is a protocol or a concrete subclass. + if not cls.__dict__.get('_is_protocol', None): + cls._is_protocol = any(b is Protocol for b in cls.__bases__) + + # Set (or override) the protocol subclass hook. + def _proto_hook(other): + if not cls.__dict__.get('_is_protocol', None): + return NotImplemented + if not getattr(cls, '_is_runtime_protocol', False): + if sys._getframe(2).f_globals['__name__'] in ['abc', 'functools']: + return NotImplemented + raise TypeError("Instance and class checks can only be used with" + " @runtime protocols") + if not _is_callable_members_only(cls): + if sys._getframe(2).f_globals['__name__'] in ['abc', 'functools']: + return NotImplemented + raise TypeError("Protocols with non-method members" + " don't support issubclass()") + if not isinstance(other, type): + # Same error as for issubclass(1, int) + raise TypeError('issubclass() arg 1 must be a class') + for attr in _get_protocol_attrs(cls): + for base in other.__mro__: + if attr in base.__dict__: + if base.__dict__[attr] is None: + return NotImplemented + break + annotations = getattr(base, '__annotations__', {}) + if (isinstance(annotations, typing.Mapping) and + attr in annotations and + isinstance(other, _ProtocolMeta) and + other._is_protocol): + break + else: + return NotImplemented + return True + if '__subclasshook__' not in cls.__dict__: + cls.__subclasshook__ = _proto_hook + + # We have nothing more to do for non-protocols. + if not cls._is_protocol: + return + + # Check consistency of bases. + for base in cls.__bases__: + if not (base in (object, typing.Generic) or + base.__module__ == 'collections.abc' and + base.__name__ in _PROTO_WHITELIST or + isinstance(base, _ProtocolMeta) and base._is_protocol): + raise TypeError('Protocols can only inherit from other' + f' protocols, got {repr(base)}') + cls.__init__ = _no_init +# 3.6 +else: + from typing import _next_in_mro, _type_check # noqa + + def _no_init(self, *args, **kwargs): + if type(self)._is_protocol: + raise TypeError('Protocols cannot be instantiated') + + class _ProtocolMeta(GenericMeta): + """Internal metaclass for Protocol. + + This exists so Protocol classes can be generic without deriving + from Generic. + """ + def __new__(cls, name, bases, namespace, + tvars=None, args=None, origin=None, extra=None, orig_bases=None): + # This is just a version copied from GenericMeta.__new__ that + # includes "Protocol" special treatment. (Comments removed for brevity.) + assert extra is None # Protocols should not have extra + if tvars is not None: + assert origin is not None + assert all(isinstance(t, typing.TypeVar) for t in tvars), tvars + else: + tvars = _type_vars(bases) + gvars = None + for base in bases: + if base is typing.Generic: + raise TypeError("Cannot inherit from plain Generic") + if (isinstance(base, GenericMeta) and + base.__origin__ in (typing.Generic, Protocol)): + if gvars is not None: + raise TypeError( + "Cannot inherit from Generic[...] or" + " Protocol[...] multiple times.") + gvars = base.__parameters__ + if gvars is None: + gvars = tvars + else: + tvarset = set(tvars) + gvarset = set(gvars) + if not tvarset <= gvarset: + s_vars = ", ".join(str(t) for t in tvars if t not in gvarset) + s_args = ", ".join(str(g) for g in gvars) + cls_name = "Generic" if any(b.__origin__ is typing.Generic + for b in bases) else "Protocol" + raise TypeError(f"Some type variables ({s_vars}) are" + f" not listed in {cls_name}[{s_args}]") + tvars = gvars + + initial_bases = bases + if (extra is not None and type(extra) is abc.ABCMeta and + extra not in bases): + bases = (extra,) + bases + bases = tuple(_gorg(b) if isinstance(b, GenericMeta) else b + for b in bases) + if any(isinstance(b, GenericMeta) and b is not typing.Generic for b in bases): + bases = tuple(b for b in bases if b is not typing.Generic) + namespace.update({'__origin__': origin, '__extra__': extra}) + self = super(GenericMeta, cls).__new__(cls, name, bases, namespace, + _root=True) + super(GenericMeta, self).__setattr__('_gorg', + self if not origin else + _gorg(origin)) + self.__parameters__ = tvars + self.__args__ = tuple(... if a is typing._TypingEllipsis else + () if a is typing._TypingEmpty else + a for a in args) if args else None + self.__next_in_mro__ = _next_in_mro(self) + if orig_bases is None: + self.__orig_bases__ = initial_bases + elif origin is not None: + self._abc_registry = origin._abc_registry + self._abc_cache = origin._abc_cache + if hasattr(self, '_subs_tree'): + self.__tree_hash__ = (hash(self._subs_tree()) if origin else + super(GenericMeta, self).__hash__()) + return self + + def __init__(cls, *args, **kwargs): + super().__init__(*args, **kwargs) + if not cls.__dict__.get('_is_protocol', None): + cls._is_protocol = any(b is Protocol or + isinstance(b, _ProtocolMeta) and + b.__origin__ is Protocol + for b in cls.__bases__) + if cls._is_protocol: + for base in cls.__mro__[1:]: + if not (base in (object, typing.Generic) or + base.__module__ == 'collections.abc' and + base.__name__ in _PROTO_WHITELIST or + isinstance(base, typing.TypingMeta) and base._is_protocol or + isinstance(base, GenericMeta) and + base.__origin__ is typing.Generic): + raise TypeError(f'Protocols can only inherit from other' + f' protocols, got {repr(base)}') + + cls.__init__ = _no_init + + def _proto_hook(other): + if not cls.__dict__.get('_is_protocol', None): + return NotImplemented + if not isinstance(other, type): + # Same error as for issubclass(1, int) + raise TypeError('issubclass() arg 1 must be a class') + for attr in _get_protocol_attrs(cls): + for base in other.__mro__: + if attr in base.__dict__: + if base.__dict__[attr] is None: + return NotImplemented + break + annotations = getattr(base, '__annotations__', {}) + if (isinstance(annotations, typing.Mapping) and + attr in annotations and + isinstance(other, _ProtocolMeta) and + other._is_protocol): + break + else: + return NotImplemented + return True + if '__subclasshook__' not in cls.__dict__: + cls.__subclasshook__ = _proto_hook + + def __instancecheck__(self, instance): + # We need this method for situations where attributes are + # assigned in __init__. + if ((not getattr(self, '_is_protocol', False) or + _is_callable_members_only(self)) and + issubclass(instance.__class__, self)): + return True + if self._is_protocol: + if all(hasattr(instance, attr) and + (not callable(getattr(self, attr, None)) or + getattr(instance, attr) is not None) + for attr in _get_protocol_attrs(self)): + return True + return super(GenericMeta, self).__instancecheck__(instance) + + def __subclasscheck__(self, cls): + if self.__origin__ is not None: + if sys._getframe(1).f_globals['__name__'] not in ['abc', 'functools']: + raise TypeError("Parameterized generics cannot be used with class " + "or instance checks") + return False + if (self.__dict__.get('_is_protocol', None) and + not self.__dict__.get('_is_runtime_protocol', None)): + if sys._getframe(1).f_globals['__name__'] in ['abc', + 'functools', + 'typing']: + return False + raise TypeError("Instance and class checks can only be used with" + " @runtime protocols") + if (self.__dict__.get('_is_runtime_protocol', None) and + not _is_callable_members_only(self)): + if sys._getframe(1).f_globals['__name__'] in ['abc', + 'functools', + 'typing']: + return super(GenericMeta, self).__subclasscheck__(cls) + raise TypeError("Protocols with non-method members" + " don't support issubclass()") + return super(GenericMeta, self).__subclasscheck__(cls) + + @typing._tp_cache + def __getitem__(self, params): + # We also need to copy this from GenericMeta.__getitem__ to get + # special treatment of "Protocol". (Comments removed for brevity.) + if not isinstance(params, tuple): + params = (params,) + if not params and _gorg(self) is not typing.Tuple: + raise TypeError( + f"Parameter list to {self.__qualname__}[...] cannot be empty") + msg = "Parameters to generic types must be types." + params = tuple(_type_check(p, msg) for p in params) + if self in (typing.Generic, Protocol): + if not all(isinstance(p, typing.TypeVar) for p in params): + raise TypeError( + f"Parameters to {repr(self)}[...] must all be type variables") + if len(set(params)) != len(params): + raise TypeError( + f"Parameters to {repr(self)}[...] must all be unique") + tvars = params + args = params + elif self in (typing.Tuple, typing.Callable): + tvars = _type_vars(params) + args = params + elif self.__origin__ in (typing.Generic, Protocol): + raise TypeError(f"Cannot subscript already-subscripted {repr(self)}") + else: + _check_generic(self, params, len(self.__parameters__)) + tvars = _type_vars(params) + args = params + + prepend = (self,) if self.__origin__ is None else () + return self.__class__(self.__name__, + prepend + self.__bases__, + _no_slots_copy(self.__dict__), + tvars=tvars, + args=args, + origin=self, + extra=self.__extra__, + orig_bases=self.__orig_bases__) + + class Protocol(metaclass=_ProtocolMeta): + """Base class for protocol classes. Protocol classes are defined as:: + + class Proto(Protocol): + def meth(self) -> int: + ... + + Such classes are primarily used with static type checkers that recognize + structural subtyping (static duck-typing), for example:: + + class C: + def meth(self) -> int: + return 0 + + def func(x: Proto) -> int: + return x.meth() + + func(C()) # Passes static type check + + See PEP 544 for details. Protocol classes decorated with + @typing_extensions.runtime act as simple-minded runtime protocol that checks + only the presence of given attributes, ignoring their type signatures. + + Protocol classes can be generic, they are defined as:: + + class GenProto(Protocol[T]): + def meth(self) -> T: + ... + """ + __slots__ = () + _is_protocol = True + + def __new__(cls, *args, **kwds): + if _gorg(cls) is Protocol: + raise TypeError("Type Protocol cannot be instantiated; " + "it can be used only as a base class") + return typing._generic_new(cls.__next_in_mro__, cls, *args, **kwds) + + +# 3.8+ +if hasattr(typing, 'runtime_checkable'): + runtime_checkable = typing.runtime_checkable +# 3.6-3.7 +else: + def runtime_checkable(cls): + """Mark a protocol class as a runtime protocol, so that it + can be used with isinstance() and issubclass(). Raise TypeError + if applied to a non-protocol class. + + This allows a simple-minded structural check very similar to the + one-offs in collections.abc such as Hashable. + """ + if not isinstance(cls, _ProtocolMeta) or not cls._is_protocol: + raise TypeError('@runtime_checkable can be only applied to protocol classes,' + f' got {cls!r}') + cls._is_runtime_protocol = True + return cls + + +# Exists for backwards compatibility. +runtime = runtime_checkable + + +# 3.8+ +if hasattr(typing, 'SupportsIndex'): + SupportsIndex = typing.SupportsIndex +# 3.6-3.7 +else: + @runtime_checkable + class SupportsIndex(Protocol): + __slots__ = () + + @abc.abstractmethod + def __index__(self) -> int: + pass + + +if hasattr(typing, "Required"): + # The standard library TypedDict in Python 3.8 does not store runtime information + # about which (if any) keys are optional. See https://bugs.python.org/issue38834 + # The standard library TypedDict in Python 3.9.0/1 does not honour the "total" + # keyword with old-style TypedDict(). See https://bugs.python.org/issue42059 + # The standard library TypedDict below Python 3.11 does not store runtime + # information about optional and required keys when using Required or NotRequired. + TypedDict = typing.TypedDict + _TypedDictMeta = typing._TypedDictMeta + is_typeddict = typing.is_typeddict +else: + def _check_fails(cls, other): + try: + if sys._getframe(1).f_globals['__name__'] not in ['abc', + 'functools', + 'typing']: + # Typed dicts are only for static structural subtyping. + raise TypeError('TypedDict does not support instance and class checks') + except (AttributeError, ValueError): + pass + return False + + def _dict_new(*args, **kwargs): + if not args: + raise TypeError('TypedDict.__new__(): not enough arguments') + _, args = args[0], args[1:] # allow the "cls" keyword be passed + return dict(*args, **kwargs) + + _dict_new.__text_signature__ = '($cls, _typename, _fields=None, /, **kwargs)' + + def _typeddict_new(*args, total=True, **kwargs): + if not args: + raise TypeError('TypedDict.__new__(): not enough arguments') + _, args = args[0], args[1:] # allow the "cls" keyword be passed + if args: + typename, args = args[0], args[1:] # allow the "_typename" keyword be passed + elif '_typename' in kwargs: + typename = kwargs.pop('_typename') + import warnings + warnings.warn("Passing '_typename' as keyword argument is deprecated", + DeprecationWarning, stacklevel=2) + else: + raise TypeError("TypedDict.__new__() missing 1 required positional " + "argument: '_typename'") + if args: + try: + fields, = args # allow the "_fields" keyword be passed + except ValueError: + raise TypeError('TypedDict.__new__() takes from 2 to 3 ' + f'positional arguments but {len(args) + 2} ' + 'were given') + elif '_fields' in kwargs and len(kwargs) == 1: + fields = kwargs.pop('_fields') + import warnings + warnings.warn("Passing '_fields' as keyword argument is deprecated", + DeprecationWarning, stacklevel=2) + else: + fields = None + + if fields is None: + fields = kwargs + elif kwargs: + raise TypeError("TypedDict takes either a dict or keyword arguments," + " but not both") + + ns = {'__annotations__': dict(fields)} + try: + # Setting correct module is necessary to make typed dict classes pickleable. + ns['__module__'] = sys._getframe(1).f_globals.get('__name__', '__main__') + except (AttributeError, ValueError): + pass + + return _TypedDictMeta(typename, (), ns, total=total) + + _typeddict_new.__text_signature__ = ('($cls, _typename, _fields=None,' + ' /, *, total=True, **kwargs)') + + class _TypedDictMeta(type): + def __init__(cls, name, bases, ns, total=True): + super().__init__(name, bases, ns) + + def __new__(cls, name, bases, ns, total=True): + # Create new typed dict class object. + # This method is called directly when TypedDict is subclassed, + # or via _typeddict_new when TypedDict is instantiated. This way + # TypedDict supports all three syntaxes described in its docstring. + # Subclasses and instances of TypedDict return actual dictionaries + # via _dict_new. + ns['__new__'] = _typeddict_new if name == 'TypedDict' else _dict_new + tp_dict = super().__new__(cls, name, (dict,), ns) + + annotations = {} + own_annotations = ns.get('__annotations__', {}) + msg = "TypedDict('Name', {f0: t0, f1: t1, ...}); each t must be a type" + own_annotations = { + n: typing._type_check(tp, msg) for n, tp in own_annotations.items() + } + required_keys = set() + optional_keys = set() + + for base in bases: + annotations.update(base.__dict__.get('__annotations__', {})) + required_keys.update(base.__dict__.get('__required_keys__', ())) + optional_keys.update(base.__dict__.get('__optional_keys__', ())) + + annotations.update(own_annotations) + if PEP_560: + for annotation_key, annotation_type in own_annotations.items(): + annotation_origin = get_origin(annotation_type) + if annotation_origin is Annotated: + annotation_args = get_args(annotation_type) + if annotation_args: + annotation_type = annotation_args[0] + annotation_origin = get_origin(annotation_type) + + if annotation_origin is Required: + required_keys.add(annotation_key) + elif annotation_origin is NotRequired: + optional_keys.add(annotation_key) + elif total: + required_keys.add(annotation_key) + else: + optional_keys.add(annotation_key) + else: + own_annotation_keys = set(own_annotations.keys()) + if total: + required_keys.update(own_annotation_keys) + else: + optional_keys.update(own_annotation_keys) + + tp_dict.__annotations__ = annotations + tp_dict.__required_keys__ = frozenset(required_keys) + tp_dict.__optional_keys__ = frozenset(optional_keys) + if not hasattr(tp_dict, '__total__'): + tp_dict.__total__ = total + return tp_dict + + __instancecheck__ = __subclasscheck__ = _check_fails + + TypedDict = _TypedDictMeta('TypedDict', (dict,), {}) + TypedDict.__module__ = __name__ + TypedDict.__doc__ = \ + """A simple typed name space. At runtime it is equivalent to a plain dict. + + TypedDict creates a dictionary type that expects all of its + instances to have a certain set of keys, with each key + associated with a value of a consistent type. This expectation + is not checked at runtime but is only enforced by type checkers. + Usage:: + + class Point2D(TypedDict): + x: int + y: int + label: str + + a: Point2D = {'x': 1, 'y': 2, 'label': 'good'} # OK + b: Point2D = {'z': 3, 'label': 'bad'} # Fails type check + + assert Point2D(x=1, y=2, label='first') == dict(x=1, y=2, label='first') + + The type info can be accessed via the Point2D.__annotations__ dict, and + the Point2D.__required_keys__ and Point2D.__optional_keys__ frozensets. + TypedDict supports two additional equivalent forms:: + + Point2D = TypedDict('Point2D', x=int, y=int, label=str) + Point2D = TypedDict('Point2D', {'x': int, 'y': int, 'label': str}) + + The class syntax is only supported in Python 3.6+, while two other + syntax forms work for Python 2.7 and 3.2+ + """ + + if hasattr(typing, "_TypedDictMeta"): + _TYPEDDICT_TYPES = (typing._TypedDictMeta, _TypedDictMeta) + else: + _TYPEDDICT_TYPES = (_TypedDictMeta,) + + def is_typeddict(tp): + """Check if an annotation is a TypedDict class + + For example:: + class Film(TypedDict): + title: str + year: int + + is_typeddict(Film) # => True + is_typeddict(Union[list, str]) # => False + """ + return isinstance(tp, tuple(_TYPEDDICT_TYPES)) + +if hasattr(typing, "Required"): + get_type_hints = typing.get_type_hints +elif PEP_560: + import functools + import types + + # replaces _strip_annotations() + def _strip_extras(t): + """Strips Annotated, Required and NotRequired from a given type.""" + if isinstance(t, _AnnotatedAlias): + return _strip_extras(t.__origin__) + if hasattr(t, "__origin__") and t.__origin__ in (Required, NotRequired): + return _strip_extras(t.__args__[0]) + if isinstance(t, typing._GenericAlias): + stripped_args = tuple(_strip_extras(a) for a in t.__args__) + if stripped_args == t.__args__: + return t + return t.copy_with(stripped_args) + if hasattr(types, "GenericAlias") and isinstance(t, types.GenericAlias): + stripped_args = tuple(_strip_extras(a) for a in t.__args__) + if stripped_args == t.__args__: + return t + return types.GenericAlias(t.__origin__, stripped_args) + if hasattr(types, "UnionType") and isinstance(t, types.UnionType): + stripped_args = tuple(_strip_extras(a) for a in t.__args__) + if stripped_args == t.__args__: + return t + return functools.reduce(operator.or_, stripped_args) + + return t + + def get_type_hints(obj, globalns=None, localns=None, include_extras=False): + """Return type hints for an object. + + This is often the same as obj.__annotations__, but it handles + forward references encoded as string literals, adds Optional[t] if a + default value equal to None is set and recursively replaces all + 'Annotated[T, ...]', 'Required[T]' or 'NotRequired[T]' with 'T' + (unless 'include_extras=True'). + + The argument may be a module, class, method, or function. The annotations + are returned as a dictionary. For classes, annotations include also + inherited members. + + TypeError is raised if the argument is not of a type that can contain + annotations, and an empty dictionary is returned if no annotations are + present. + + BEWARE -- the behavior of globalns and localns is counterintuitive + (unless you are familiar with how eval() and exec() work). The + search order is locals first, then globals. + + - If no dict arguments are passed, an attempt is made to use the + globals from obj (or the respective module's globals for classes), + and these are also used as the locals. If the object does not appear + to have globals, an empty dictionary is used. + + - If one dict argument is passed, it is used for both globals and + locals. + + - If two dict arguments are passed, they specify globals and + locals, respectively. + """ + if hasattr(typing, "Annotated"): + hint = typing.get_type_hints( + obj, globalns=globalns, localns=localns, include_extras=True + ) + else: + hint = typing.get_type_hints(obj, globalns=globalns, localns=localns) + if include_extras: + return hint + return {k: _strip_extras(t) for k, t in hint.items()} + + +# Python 3.9+ has PEP 593 (Annotated) +if hasattr(typing, 'Annotated'): + Annotated = typing.Annotated + # Not exported and not a public API, but needed for get_origin() and get_args() + # to work. + _AnnotatedAlias = typing._AnnotatedAlias +# 3.7-3.8 +elif PEP_560: + class _AnnotatedAlias(typing._GenericAlias, _root=True): + """Runtime representation of an annotated type. + + At its core 'Annotated[t, dec1, dec2, ...]' is an alias for the type 't' + with extra annotations. The alias behaves like a normal typing alias, + instantiating is the same as instantiating the underlying type, binding + it to types is also the same. + """ + def __init__(self, origin, metadata): + if isinstance(origin, _AnnotatedAlias): + metadata = origin.__metadata__ + metadata + origin = origin.__origin__ + super().__init__(origin, origin) + self.__metadata__ = metadata + + def copy_with(self, params): + assert len(params) == 1 + new_type = params[0] + return _AnnotatedAlias(new_type, self.__metadata__) + + def __repr__(self): + return (f"typing_extensions.Annotated[{typing._type_repr(self.__origin__)}, " + f"{', '.join(repr(a) for a in self.__metadata__)}]") + + def __reduce__(self): + return operator.getitem, ( + Annotated, (self.__origin__,) + self.__metadata__ + ) + + def __eq__(self, other): + if not isinstance(other, _AnnotatedAlias): + return NotImplemented + if self.__origin__ != other.__origin__: + return False + return self.__metadata__ == other.__metadata__ + + def __hash__(self): + return hash((self.__origin__, self.__metadata__)) + + class Annotated: + """Add context specific metadata to a type. + + Example: Annotated[int, runtime_check.Unsigned] indicates to the + hypothetical runtime_check module that this type is an unsigned int. + Every other consumer of this type can ignore this metadata and treat + this type as int. + + The first argument to Annotated must be a valid type (and will be in + the __origin__ field), the remaining arguments are kept as a tuple in + the __extra__ field. + + Details: + + - It's an error to call `Annotated` with less than two arguments. + - Nested Annotated are flattened:: + + Annotated[Annotated[T, Ann1, Ann2], Ann3] == Annotated[T, Ann1, Ann2, Ann3] + + - Instantiating an annotated type is equivalent to instantiating the + underlying type:: + + Annotated[C, Ann1](5) == C(5) + + - Annotated can be used as a generic type alias:: + + Optimized = Annotated[T, runtime.Optimize()] + Optimized[int] == Annotated[int, runtime.Optimize()] + + OptimizedList = Annotated[List[T], runtime.Optimize()] + OptimizedList[int] == Annotated[List[int], runtime.Optimize()] + """ + + __slots__ = () + + def __new__(cls, *args, **kwargs): + raise TypeError("Type Annotated cannot be instantiated.") + + @typing._tp_cache + def __class_getitem__(cls, params): + if not isinstance(params, tuple) or len(params) < 2: + raise TypeError("Annotated[...] should be used " + "with at least two arguments (a type and an " + "annotation).") + allowed_special_forms = (ClassVar, Final) + if get_origin(params[0]) in allowed_special_forms: + origin = params[0] + else: + msg = "Annotated[t, ...]: t must be a type." + origin = typing._type_check(params[0], msg) + metadata = tuple(params[1:]) + return _AnnotatedAlias(origin, metadata) + + def __init_subclass__(cls, *args, **kwargs): + raise TypeError( + f"Cannot subclass {cls.__module__}.Annotated" + ) +# 3.6 +else: + + def _is_dunder(name): + """Returns True if name is a __dunder_variable_name__.""" + return len(name) > 4 and name.startswith('__') and name.endswith('__') + + # Prior to Python 3.7 types did not have `copy_with`. A lot of the equality + # checks, argument expansion etc. are done on the _subs_tre. As a result we + # can't provide a get_type_hints function that strips out annotations. + + class AnnotatedMeta(typing.GenericMeta): + """Metaclass for Annotated""" + + def __new__(cls, name, bases, namespace, **kwargs): + if any(b is not object for b in bases): + raise TypeError("Cannot subclass " + str(Annotated)) + return super().__new__(cls, name, bases, namespace, **kwargs) + + @property + def __metadata__(self): + return self._subs_tree()[2] + + def _tree_repr(self, tree): + cls, origin, metadata = tree + if not isinstance(origin, tuple): + tp_repr = typing._type_repr(origin) + else: + tp_repr = origin[0]._tree_repr(origin) + metadata_reprs = ", ".join(repr(arg) for arg in metadata) + return f'{cls}[{tp_repr}, {metadata_reprs}]' + + def _subs_tree(self, tvars=None, args=None): # noqa + if self is Annotated: + return Annotated + res = super()._subs_tree(tvars=tvars, args=args) + # Flatten nested Annotated + if isinstance(res[1], tuple) and res[1][0] is Annotated: + sub_tp = res[1][1] + sub_annot = res[1][2] + return (Annotated, sub_tp, sub_annot + res[2]) + return res + + def _get_cons(self): + """Return the class used to create instance of this type.""" + if self.__origin__ is None: + raise TypeError("Cannot get the underlying type of a " + "non-specialized Annotated type.") + tree = self._subs_tree() + while isinstance(tree, tuple) and tree[0] is Annotated: + tree = tree[1] + if isinstance(tree, tuple): + return tree[0] + else: + return tree + + @typing._tp_cache + def __getitem__(self, params): + if not isinstance(params, tuple): + params = (params,) + if self.__origin__ is not None: # specializing an instantiated type + return super().__getitem__(params) + elif not isinstance(params, tuple) or len(params) < 2: + raise TypeError("Annotated[...] should be instantiated " + "with at least two arguments (a type and an " + "annotation).") + else: + if ( + isinstance(params[0], typing._TypingBase) and + type(params[0]).__name__ == "_ClassVar" + ): + tp = params[0] + else: + msg = "Annotated[t, ...]: t must be a type." + tp = typing._type_check(params[0], msg) + metadata = tuple(params[1:]) + return self.__class__( + self.__name__, + self.__bases__, + _no_slots_copy(self.__dict__), + tvars=_type_vars((tp,)), + # Metadata is a tuple so it won't be touched by _replace_args et al. + args=(tp, metadata), + origin=self, + ) + + def __call__(self, *args, **kwargs): + cons = self._get_cons() + result = cons(*args, **kwargs) + try: + result.__orig_class__ = self + except AttributeError: + pass + return result + + def __getattr__(self, attr): + # For simplicity we just don't relay all dunder names + if self.__origin__ is not None and not _is_dunder(attr): + return getattr(self._get_cons(), attr) + raise AttributeError(attr) + + def __setattr__(self, attr, value): + if _is_dunder(attr) or attr.startswith('_abc_'): + super().__setattr__(attr, value) + elif self.__origin__ is None: + raise AttributeError(attr) + else: + setattr(self._get_cons(), attr, value) + + def __instancecheck__(self, obj): + raise TypeError("Annotated cannot be used with isinstance().") + + def __subclasscheck__(self, cls): + raise TypeError("Annotated cannot be used with issubclass().") + + class Annotated(metaclass=AnnotatedMeta): + """Add context specific metadata to a type. + + Example: Annotated[int, runtime_check.Unsigned] indicates to the + hypothetical runtime_check module that this type is an unsigned int. + Every other consumer of this type can ignore this metadata and treat + this type as int. + + The first argument to Annotated must be a valid type, the remaining + arguments are kept as a tuple in the __metadata__ field. + + Details: + + - It's an error to call `Annotated` with less than two arguments. + - Nested Annotated are flattened:: + + Annotated[Annotated[T, Ann1, Ann2], Ann3] == Annotated[T, Ann1, Ann2, Ann3] + + - Instantiating an annotated type is equivalent to instantiating the + underlying type:: + + Annotated[C, Ann1](5) == C(5) + + - Annotated can be used as a generic type alias:: + + Optimized = Annotated[T, runtime.Optimize()] + Optimized[int] == Annotated[int, runtime.Optimize()] + + OptimizedList = Annotated[List[T], runtime.Optimize()] + OptimizedList[int] == Annotated[List[int], runtime.Optimize()] + """ + +# Python 3.8 has get_origin() and get_args() but those implementations aren't +# Annotated-aware, so we can't use those. Python 3.9's versions don't support +# ParamSpecArgs and ParamSpecKwargs, so only Python 3.10's versions will do. +if sys.version_info[:2] >= (3, 10): + get_origin = typing.get_origin + get_args = typing.get_args +# 3.7-3.9 +elif PEP_560: + try: + # 3.9+ + from typing import _BaseGenericAlias + except ImportError: + _BaseGenericAlias = typing._GenericAlias + try: + # 3.9+ + from typing import GenericAlias + except ImportError: + GenericAlias = typing._GenericAlias + + def get_origin(tp): + """Get the unsubscripted version of a type. + + This supports generic types, Callable, Tuple, Union, Literal, Final, ClassVar + and Annotated. Return None for unsupported types. Examples:: + + get_origin(Literal[42]) is Literal + get_origin(int) is None + get_origin(ClassVar[int]) is ClassVar + get_origin(Generic) is Generic + get_origin(Generic[T]) is Generic + get_origin(Union[T, int]) is Union + get_origin(List[Tuple[T, T]][int]) == list + get_origin(P.args) is P + """ + if isinstance(tp, _AnnotatedAlias): + return Annotated + if isinstance(tp, (typing._GenericAlias, GenericAlias, _BaseGenericAlias, + ParamSpecArgs, ParamSpecKwargs)): + return tp.__origin__ + if tp is typing.Generic: + return typing.Generic + return None + + def get_args(tp): + """Get type arguments with all substitutions performed. + + For unions, basic simplifications used by Union constructor are performed. + Examples:: + get_args(Dict[str, int]) == (str, int) + get_args(int) == () + get_args(Union[int, Union[T, int], str][int]) == (int, str) + get_args(Union[int, Tuple[T, int]][str]) == (int, Tuple[str, int]) + get_args(Callable[[], T][int]) == ([], int) + """ + if isinstance(tp, _AnnotatedAlias): + return (tp.__origin__,) + tp.__metadata__ + if isinstance(tp, (typing._GenericAlias, GenericAlias)): + if getattr(tp, "_special", False): + return () + res = tp.__args__ + if get_origin(tp) is collections.abc.Callable and res[0] is not Ellipsis: + res = (list(res[:-1]), res[-1]) + return res + return () + + +# 3.10+ +if hasattr(typing, 'TypeAlias'): + TypeAlias = typing.TypeAlias +# 3.9 +elif sys.version_info[:2] >= (3, 9): + class _TypeAliasForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + @_TypeAliasForm + def TypeAlias(self, parameters): + """Special marker indicating that an assignment should + be recognized as a proper type alias definition by type + checkers. + + For example:: + + Predicate: TypeAlias = Callable[..., bool] + + It's invalid when used anywhere except as in the example above. + """ + raise TypeError(f"{self} is not subscriptable") +# 3.7-3.8 +elif sys.version_info[:2] >= (3, 7): + class _TypeAliasForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + TypeAlias = _TypeAliasForm('TypeAlias', + doc="""Special marker indicating that an assignment should + be recognized as a proper type alias definition by type + checkers. + + For example:: + + Predicate: TypeAlias = Callable[..., bool] + + It's invalid when used anywhere except as in the example + above.""") +# 3.6 +else: + class _TypeAliasMeta(typing.TypingMeta): + """Metaclass for TypeAlias""" + + def __repr__(self): + return 'typing_extensions.TypeAlias' + + class _TypeAliasBase(typing._FinalTypingBase, metaclass=_TypeAliasMeta, _root=True): + """Special marker indicating that an assignment should + be recognized as a proper type alias definition by type + checkers. + + For example:: + + Predicate: TypeAlias = Callable[..., bool] + + It's invalid when used anywhere except as in the example above. + """ + __slots__ = () + + def __instancecheck__(self, obj): + raise TypeError("TypeAlias cannot be used with isinstance().") + + def __subclasscheck__(self, cls): + raise TypeError("TypeAlias cannot be used with issubclass().") + + def __repr__(self): + return 'typing_extensions.TypeAlias' + + TypeAlias = _TypeAliasBase(_root=True) + + +# Python 3.10+ has PEP 612 +if hasattr(typing, 'ParamSpecArgs'): + ParamSpecArgs = typing.ParamSpecArgs + ParamSpecKwargs = typing.ParamSpecKwargs +# 3.6-3.9 +else: + class _Immutable: + """Mixin to indicate that object should not be copied.""" + __slots__ = () + + def __copy__(self): + return self + + def __deepcopy__(self, memo): + return self + + class ParamSpecArgs(_Immutable): + """The args for a ParamSpec object. + + Given a ParamSpec object P, P.args is an instance of ParamSpecArgs. + + ParamSpecArgs objects have a reference back to their ParamSpec: + + P.args.__origin__ is P + + This type is meant for runtime introspection and has no special meaning to + static type checkers. + """ + def __init__(self, origin): + self.__origin__ = origin + + def __repr__(self): + return f"{self.__origin__.__name__}.args" + + def __eq__(self, other): + if not isinstance(other, ParamSpecArgs): + return NotImplemented + return self.__origin__ == other.__origin__ + + class ParamSpecKwargs(_Immutable): + """The kwargs for a ParamSpec object. + + Given a ParamSpec object P, P.kwargs is an instance of ParamSpecKwargs. + + ParamSpecKwargs objects have a reference back to their ParamSpec: + + P.kwargs.__origin__ is P + + This type is meant for runtime introspection and has no special meaning to + static type checkers. + """ + def __init__(self, origin): + self.__origin__ = origin + + def __repr__(self): + return f"{self.__origin__.__name__}.kwargs" + + def __eq__(self, other): + if not isinstance(other, ParamSpecKwargs): + return NotImplemented + return self.__origin__ == other.__origin__ + +# 3.10+ +if hasattr(typing, 'ParamSpec'): + ParamSpec = typing.ParamSpec +# 3.6-3.9 +else: + + # Inherits from list as a workaround for Callable checks in Python < 3.9.2. + class ParamSpec(list): + """Parameter specification variable. + + Usage:: + + P = ParamSpec('P') + + Parameter specification variables exist primarily for the benefit of static + type checkers. They are used to forward the parameter types of one + callable to another callable, a pattern commonly found in higher order + functions and decorators. They are only valid when used in ``Concatenate``, + or s the first argument to ``Callable``. In Python 3.10 and higher, + they are also supported in user-defined Generics at runtime. + See class Generic for more information on generic types. An + example for annotating a decorator:: + + T = TypeVar('T') + P = ParamSpec('P') + + def add_logging(f: Callable[P, T]) -> Callable[P, T]: + '''A type-safe decorator to add logging to a function.''' + def inner(*args: P.args, **kwargs: P.kwargs) -> T: + logging.info(f'{f.__name__} was called') + return f(*args, **kwargs) + return inner + + @add_logging + def add_two(x: float, y: float) -> float: + '''Add two numbers together.''' + return x + y + + Parameter specification variables defined with covariant=True or + contravariant=True can be used to declare covariant or contravariant + generic types. These keyword arguments are valid, but their actual semantics + are yet to be decided. See PEP 612 for details. + + Parameter specification variables can be introspected. e.g.: + + P.__name__ == 'T' + P.__bound__ == None + P.__covariant__ == False + P.__contravariant__ == False + + Note that only parameter specification variables defined in global scope can + be pickled. + """ + + # Trick Generic __parameters__. + __class__ = typing.TypeVar + + @property + def args(self): + return ParamSpecArgs(self) + + @property + def kwargs(self): + return ParamSpecKwargs(self) + + def __init__(self, name, *, bound=None, covariant=False, contravariant=False): + super().__init__([self]) + self.__name__ = name + self.__covariant__ = bool(covariant) + self.__contravariant__ = bool(contravariant) + if bound: + self.__bound__ = typing._type_check(bound, 'Bound must be a type.') + else: + self.__bound__ = None + + # for pickling: + try: + def_mod = sys._getframe(1).f_globals.get('__name__', '__main__') + except (AttributeError, ValueError): + def_mod = None + if def_mod != 'typing_extensions': + self.__module__ = def_mod + + def __repr__(self): + if self.__covariant__: + prefix = '+' + elif self.__contravariant__: + prefix = '-' + else: + prefix = '~' + return prefix + self.__name__ + + def __hash__(self): + return object.__hash__(self) + + def __eq__(self, other): + return self is other + + def __reduce__(self): + return self.__name__ + + # Hack to get typing._type_check to pass. + def __call__(self, *args, **kwargs): + pass + + if not PEP_560: + # Only needed in 3.6. + def _get_type_vars(self, tvars): + if self not in tvars: + tvars.append(self) + + +# 3.6-3.9 +if not hasattr(typing, 'Concatenate'): + # Inherits from list as a workaround for Callable checks in Python < 3.9.2. + class _ConcatenateGenericAlias(list): + + # Trick Generic into looking into this for __parameters__. + if PEP_560: + __class__ = typing._GenericAlias + else: + __class__ = typing._TypingBase + + # Flag in 3.8. + _special = False + # Attribute in 3.6 and earlier. + _gorg = typing.Generic + + def __init__(self, origin, args): + super().__init__(args) + self.__origin__ = origin + self.__args__ = args + + def __repr__(self): + _type_repr = typing._type_repr + return (f'{_type_repr(self.__origin__)}' + f'[{", ".join(_type_repr(arg) for arg in self.__args__)}]') + + def __hash__(self): + return hash((self.__origin__, self.__args__)) + + # Hack to get typing._type_check to pass in Generic. + def __call__(self, *args, **kwargs): + pass + + @property + def __parameters__(self): + return tuple( + tp for tp in self.__args__ if isinstance(tp, (typing.TypeVar, ParamSpec)) + ) + + if not PEP_560: + # Only required in 3.6. + def _get_type_vars(self, tvars): + if self.__origin__ and self.__parameters__: + typing._get_type_vars(self.__parameters__, tvars) + + +# 3.6-3.9 +@typing._tp_cache +def _concatenate_getitem(self, parameters): + if parameters == (): + raise TypeError("Cannot take a Concatenate of no types.") + if not isinstance(parameters, tuple): + parameters = (parameters,) + if not isinstance(parameters[-1], ParamSpec): + raise TypeError("The last parameter to Concatenate should be a " + "ParamSpec variable.") + msg = "Concatenate[arg, ...]: each arg must be a type." + parameters = tuple(typing._type_check(p, msg) for p in parameters) + return _ConcatenateGenericAlias(self, parameters) + + +# 3.10+ +if hasattr(typing, 'Concatenate'): + Concatenate = typing.Concatenate + _ConcatenateGenericAlias = typing._ConcatenateGenericAlias # noqa +# 3.9 +elif sys.version_info[:2] >= (3, 9): + @_TypeAliasForm + def Concatenate(self, parameters): + """Used in conjunction with ``ParamSpec`` and ``Callable`` to represent a + higher order function which adds, removes or transforms parameters of a + callable. + + For example:: + + Callable[Concatenate[int, P], int] + + See PEP 612 for detailed information. + """ + return _concatenate_getitem(self, parameters) +# 3.7-8 +elif sys.version_info[:2] >= (3, 7): + class _ConcatenateForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + def __getitem__(self, parameters): + return _concatenate_getitem(self, parameters) + + Concatenate = _ConcatenateForm( + 'Concatenate', + doc="""Used in conjunction with ``ParamSpec`` and ``Callable`` to represent a + higher order function which adds, removes or transforms parameters of a + callable. + + For example:: + + Callable[Concatenate[int, P], int] + + See PEP 612 for detailed information. + """) +# 3.6 +else: + class _ConcatenateAliasMeta(typing.TypingMeta): + """Metaclass for Concatenate.""" + + def __repr__(self): + return 'typing_extensions.Concatenate' + + class _ConcatenateAliasBase(typing._FinalTypingBase, + metaclass=_ConcatenateAliasMeta, + _root=True): + """Used in conjunction with ``ParamSpec`` and ``Callable`` to represent a + higher order function which adds, removes or transforms parameters of a + callable. + + For example:: + + Callable[Concatenate[int, P], int] + + See PEP 612 for detailed information. + """ + __slots__ = () + + def __instancecheck__(self, obj): + raise TypeError("Concatenate cannot be used with isinstance().") + + def __subclasscheck__(self, cls): + raise TypeError("Concatenate cannot be used with issubclass().") + + def __repr__(self): + return 'typing_extensions.Concatenate' + + def __getitem__(self, parameters): + return _concatenate_getitem(self, parameters) + + Concatenate = _ConcatenateAliasBase(_root=True) + +# 3.10+ +if hasattr(typing, 'TypeGuard'): + TypeGuard = typing.TypeGuard +# 3.9 +elif sys.version_info[:2] >= (3, 9): + class _TypeGuardForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + @_TypeGuardForm + def TypeGuard(self, parameters): + """Special typing form used to annotate the return type of a user-defined + type guard function. ``TypeGuard`` only accepts a single type argument. + At runtime, functions marked this way should return a boolean. + + ``TypeGuard`` aims to benefit *type narrowing* -- a technique used by static + type checkers to determine a more precise type of an expression within a + program's code flow. Usually type narrowing is done by analyzing + conditional code flow and applying the narrowing to a block of code. The + conditional expression here is sometimes referred to as a "type guard". + + Sometimes it would be convenient to use a user-defined boolean function + as a type guard. Such a function should use ``TypeGuard[...]`` as its + return type to alert static type checkers to this intention. + + Using ``-> TypeGuard`` tells the static type checker that for a given + function: + + 1. The return value is a boolean. + 2. If the return value is ``True``, the type of its argument + is the type inside ``TypeGuard``. + + For example:: + + def is_str(val: Union[str, float]): + # "isinstance" type guard + if isinstance(val, str): + # Type of ``val`` is narrowed to ``str`` + ... + else: + # Else, type of ``val`` is narrowed to ``float``. + ... + + Strict type narrowing is not enforced -- ``TypeB`` need not be a narrower + form of ``TypeA`` (it can even be a wider form) and this may lead to + type-unsafe results. The main reason is to allow for things like + narrowing ``List[object]`` to ``List[str]`` even though the latter is not + a subtype of the former, since ``List`` is invariant. The responsibility of + writing type-safe type guards is left to the user. + + ``TypeGuard`` also works with type variables. For more information, see + PEP 647 (User-Defined Type Guards). + """ + item = typing._type_check(parameters, f'{self} accepts only single type.') + return typing._GenericAlias(self, (item,)) +# 3.7-3.8 +elif sys.version_info[:2] >= (3, 7): + class _TypeGuardForm(typing._SpecialForm, _root=True): + + def __repr__(self): + return 'typing_extensions.' + self._name + + def __getitem__(self, parameters): + item = typing._type_check(parameters, + f'{self._name} accepts only a single type') + return typing._GenericAlias(self, (item,)) + + TypeGuard = _TypeGuardForm( + 'TypeGuard', + doc="""Special typing form used to annotate the return type of a user-defined + type guard function. ``TypeGuard`` only accepts a single type argument. + At runtime, functions marked this way should return a boolean. + + ``TypeGuard`` aims to benefit *type narrowing* -- a technique used by static + type checkers to determine a more precise type of an expression within a + program's code flow. Usually type narrowing is done by analyzing + conditional code flow and applying the narrowing to a block of code. The + conditional expression here is sometimes referred to as a "type guard". + + Sometimes it would be convenient to use a user-defined boolean function + as a type guard. Such a function should use ``TypeGuard[...]`` as its + return type to alert static type checkers to this intention. + + Using ``-> TypeGuard`` tells the static type checker that for a given + function: + + 1. The return value is a boolean. + 2. If the return value is ``True``, the type of its argument + is the type inside ``TypeGuard``. + + For example:: + + def is_str(val: Union[str, float]): + # "isinstance" type guard + if isinstance(val, str): + # Type of ``val`` is narrowed to ``str`` + ... + else: + # Else, type of ``val`` is narrowed to ``float``. + ... + + Strict type narrowing is not enforced -- ``TypeB`` need not be a narrower + form of ``TypeA`` (it can even be a wider form) and this may lead to + type-unsafe results. The main reason is to allow for things like + narrowing ``List[object]`` to ``List[str]`` even though the latter is not + a subtype of the former, since ``List`` is invariant. The responsibility of + writing type-safe type guards is left to the user. + + ``TypeGuard`` also works with type variables. For more information, see + PEP 647 (User-Defined Type Guards). + """) +# 3.6 +else: + class _TypeGuard(typing._FinalTypingBase, _root=True): + """Special typing form used to annotate the return type of a user-defined + type guard function. ``TypeGuard`` only accepts a single type argument. + At runtime, functions marked this way should return a boolean. + + ``TypeGuard`` aims to benefit *type narrowing* -- a technique used by static + type checkers to determine a more precise type of an expression within a + program's code flow. Usually type narrowing is done by analyzing + conditional code flow and applying the narrowing to a block of code. The + conditional expression here is sometimes referred to as a "type guard". + + Sometimes it would be convenient to use a user-defined boolean function + as a type guard. Such a function should use ``TypeGuard[...]`` as its + return type to alert static type checkers to this intention. + + Using ``-> TypeGuard`` tells the static type checker that for a given + function: + + 1. The return value is a boolean. + 2. If the return value is ``True``, the type of its argument + is the type inside ``TypeGuard``. + + For example:: + + def is_str(val: Union[str, float]): + # "isinstance" type guard + if isinstance(val, str): + # Type of ``val`` is narrowed to ``str`` + ... + else: + # Else, type of ``val`` is narrowed to ``float``. + ... + + Strict type narrowing is not enforced -- ``TypeB`` need not be a narrower + form of ``TypeA`` (it can even be a wider form) and this may lead to + type-unsafe results. The main reason is to allow for things like + narrowing ``List[object]`` to ``List[str]`` even though the latter is not + a subtype of the former, since ``List`` is invariant. The responsibility of + writing type-safe type guards is left to the user. + + ``TypeGuard`` also works with type variables. For more information, see + PEP 647 (User-Defined Type Guards). + """ + + __slots__ = ('__type__',) + + def __init__(self, tp=None, **kwds): + self.__type__ = tp + + def __getitem__(self, item): + cls = type(self) + if self.__type__ is None: + return cls(typing._type_check(item, + f'{cls.__name__[1:]} accepts only a single type.'), + _root=True) + raise TypeError(f'{cls.__name__[1:]} cannot be further subscripted') + + def _eval_type(self, globalns, localns): + new_tp = typing._eval_type(self.__type__, globalns, localns) + if new_tp == self.__type__: + return self + return type(self)(new_tp, _root=True) + + def __repr__(self): + r = super().__repr__() + if self.__type__ is not None: + r += f'[{typing._type_repr(self.__type__)}]' + return r + + def __hash__(self): + return hash((type(self).__name__, self.__type__)) + + def __eq__(self, other): + if not isinstance(other, _TypeGuard): + return NotImplemented + if self.__type__ is not None: + return self.__type__ == other.__type__ + return self is other + + TypeGuard = _TypeGuard(_root=True) + + +if sys.version_info[:2] >= (3, 7): + # Vendored from cpython typing._SpecialFrom + class _SpecialForm(typing._Final, _root=True): + __slots__ = ('_name', '__doc__', '_getitem') + + def __init__(self, getitem): + self._getitem = getitem + self._name = getitem.__name__ + self.__doc__ = getitem.__doc__ + + def __getattr__(self, item): + if item in {'__name__', '__qualname__'}: + return self._name + + raise AttributeError(item) + + def __mro_entries__(self, bases): + raise TypeError(f"Cannot subclass {self!r}") + + def __repr__(self): + return f'typing_extensions.{self._name}' + + def __reduce__(self): + return self._name + + def __call__(self, *args, **kwds): + raise TypeError(f"Cannot instantiate {self!r}") + + def __or__(self, other): + return typing.Union[self, other] + + def __ror__(self, other): + return typing.Union[other, self] + + def __instancecheck__(self, obj): + raise TypeError(f"{self} cannot be used with isinstance()") + + def __subclasscheck__(self, cls): + raise TypeError(f"{self} cannot be used with issubclass()") + + @typing._tp_cache + def __getitem__(self, parameters): + return self._getitem(self, parameters) + + +if hasattr(typing, "LiteralString"): + LiteralString = typing.LiteralString +elif sys.version_info[:2] >= (3, 7): + @_SpecialForm + def LiteralString(self, params): + """Represents an arbitrary literal string. + + Example:: + + from metaflow._vendor.v3_6.typing_extensions import LiteralString + + def query(sql: LiteralString) -> ...: + ... + + query("SELECT * FROM table") # ok + query(f"SELECT * FROM {input()}") # not ok + + See PEP 675 for details. + + """ + raise TypeError(f"{self} is not subscriptable") +else: + class _LiteralString(typing._FinalTypingBase, _root=True): + """Represents an arbitrary literal string. + + Example:: + + from metaflow._vendor.v3_6.typing_extensions import LiteralString + + def query(sql: LiteralString) -> ...: + ... + + query("SELECT * FROM table") # ok + query(f"SELECT * FROM {input()}") # not ok + + See PEP 675 for details. + + """ + + __slots__ = () + + def __instancecheck__(self, obj): + raise TypeError(f"{self} cannot be used with isinstance().") + + def __subclasscheck__(self, cls): + raise TypeError(f"{self} cannot be used with issubclass().") + + LiteralString = _LiteralString(_root=True) + + +if hasattr(typing, "Self"): + Self = typing.Self +elif sys.version_info[:2] >= (3, 7): + @_SpecialForm + def Self(self, params): + """Used to spell the type of "self" in classes. + + Example:: + + from typing import Self + + class ReturnsSelf: + def parse(self, data: bytes) -> Self: + ... + return self + + """ + + raise TypeError(f"{self} is not subscriptable") +else: + class _Self(typing._FinalTypingBase, _root=True): + """Used to spell the type of "self" in classes. + + Example:: + + from typing import Self + + class ReturnsSelf: + def parse(self, data: bytes) -> Self: + ... + return self + + """ + + __slots__ = () + + def __instancecheck__(self, obj): + raise TypeError(f"{self} cannot be used with isinstance().") + + def __subclasscheck__(self, cls): + raise TypeError(f"{self} cannot be used with issubclass().") + + Self = _Self(_root=True) + + +if hasattr(typing, "Never"): + Never = typing.Never +elif sys.version_info[:2] >= (3, 7): + @_SpecialForm + def Never(self, params): + """The bottom type, a type that has no members. + + This can be used to define a function that should never be + called, or a function that never returns:: + + from metaflow._vendor.v3_6.typing_extensions import Never + + def never_call_me(arg: Never) -> None: + pass + + def int_or_str(arg: int | str) -> None: + never_call_me(arg) # type checker error + match arg: + case int(): + print("It's an int") + case str(): + print("It's a str") + case _: + never_call_me(arg) # ok, arg is of type Never + + """ + + raise TypeError(f"{self} is not subscriptable") +else: + class _Never(typing._FinalTypingBase, _root=True): + """The bottom type, a type that has no members. + + This can be used to define a function that should never be + called, or a function that never returns:: + + from metaflow._vendor.v3_6.typing_extensions import Never + + def never_call_me(arg: Never) -> None: + pass + + def int_or_str(arg: int | str) -> None: + never_call_me(arg) # type checker error + match arg: + case int(): + print("It's an int") + case str(): + print("It's a str") + case _: + never_call_me(arg) # ok, arg is of type Never + + """ + + __slots__ = () + + def __instancecheck__(self, obj): + raise TypeError(f"{self} cannot be used with isinstance().") + + def __subclasscheck__(self, cls): + raise TypeError(f"{self} cannot be used with issubclass().") + + Never = _Never(_root=True) + + +if hasattr(typing, 'Required'): + Required = typing.Required + NotRequired = typing.NotRequired +elif sys.version_info[:2] >= (3, 9): + class _ExtensionsSpecialForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + @_ExtensionsSpecialForm + def Required(self, parameters): + """A special typing construct to mark a key of a total=False TypedDict + as required. For example: + + class Movie(TypedDict, total=False): + title: Required[str] + year: int + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + + There is no runtime checking that a required key is actually provided + when instantiating a related TypedDict. + """ + item = typing._type_check(parameters, f'{self._name} accepts only single type') + return typing._GenericAlias(self, (item,)) + + @_ExtensionsSpecialForm + def NotRequired(self, parameters): + """A special typing construct to mark a key of a TypedDict as + potentially missing. For example: + + class Movie(TypedDict): + title: str + year: NotRequired[int] + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + """ + item = typing._type_check(parameters, f'{self._name} accepts only single type') + return typing._GenericAlias(self, (item,)) + +elif sys.version_info[:2] >= (3, 7): + class _RequiredForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + def __getitem__(self, parameters): + item = typing._type_check(parameters, + '{} accepts only single type'.format(self._name)) + return typing._GenericAlias(self, (item,)) + + Required = _RequiredForm( + 'Required', + doc="""A special typing construct to mark a key of a total=False TypedDict + as required. For example: + + class Movie(TypedDict, total=False): + title: Required[str] + year: int + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + + There is no runtime checking that a required key is actually provided + when instantiating a related TypedDict. + """) + NotRequired = _RequiredForm( + 'NotRequired', + doc="""A special typing construct to mark a key of a TypedDict as + potentially missing. For example: + + class Movie(TypedDict): + title: str + year: NotRequired[int] + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + """) +else: + # NOTE: Modeled after _Final's implementation when _FinalTypingBase available + class _MaybeRequired(typing._FinalTypingBase, _root=True): + __slots__ = ('__type__',) + + def __init__(self, tp=None, **kwds): + self.__type__ = tp + + def __getitem__(self, item): + cls = type(self) + if self.__type__ is None: + return cls(typing._type_check(item, + '{} accepts only single type.'.format(cls.__name__[1:])), + _root=True) + raise TypeError('{} cannot be further subscripted' + .format(cls.__name__[1:])) + + def _eval_type(self, globalns, localns): + new_tp = typing._eval_type(self.__type__, globalns, localns) + if new_tp == self.__type__: + return self + return type(self)(new_tp, _root=True) + + def __repr__(self): + r = super().__repr__() + if self.__type__ is not None: + r += '[{}]'.format(typing._type_repr(self.__type__)) + return r + + def __hash__(self): + return hash((type(self).__name__, self.__type__)) + + def __eq__(self, other): + if not isinstance(other, type(self)): + return NotImplemented + if self.__type__ is not None: + return self.__type__ == other.__type__ + return self is other + + class _Required(_MaybeRequired, _root=True): + """A special typing construct to mark a key of a total=False TypedDict + as required. For example: + + class Movie(TypedDict, total=False): + title: Required[str] + year: int + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + + There is no runtime checking that a required key is actually provided + when instantiating a related TypedDict. + """ + + class _NotRequired(_MaybeRequired, _root=True): + """A special typing construct to mark a key of a TypedDict as + potentially missing. For example: + + class Movie(TypedDict): + title: str + year: NotRequired[int] + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + """ + + Required = _Required(_root=True) + NotRequired = _NotRequired(_root=True) + + +if sys.version_info[:2] >= (3, 9): + class _UnpackSpecialForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + class _UnpackAlias(typing._GenericAlias, _root=True): + __class__ = typing.TypeVar + + @_UnpackSpecialForm + def Unpack(self, parameters): + """A special typing construct to unpack a variadic type. For example: + + Shape = TypeVarTuple('Shape') + Batch = NewType('Batch', int) + + def add_batch_axis( + x: Array[Unpack[Shape]] + ) -> Array[Batch, Unpack[Shape]]: ... + + """ + item = typing._type_check(parameters, f'{self._name} accepts only single type') + return _UnpackAlias(self, (item,)) + + def _is_unpack(obj): + return isinstance(obj, _UnpackAlias) + +elif sys.version_info[:2] >= (3, 7): + class _UnpackAlias(typing._GenericAlias, _root=True): + __class__ = typing.TypeVar + + class _UnpackForm(typing._SpecialForm, _root=True): + def __repr__(self): + return 'typing_extensions.' + self._name + + def __getitem__(self, parameters): + item = typing._type_check(parameters, + f'{self._name} accepts only single type') + return _UnpackAlias(self, (item,)) + + Unpack = _UnpackForm( + 'Unpack', + doc="""A special typing construct to unpack a variadic type. For example: + + Shape = TypeVarTuple('Shape') + Batch = NewType('Batch', int) + + def add_batch_axis( + x: Array[Unpack[Shape]] + ) -> Array[Batch, Unpack[Shape]]: ... + + """) + + def _is_unpack(obj): + return isinstance(obj, _UnpackAlias) + +else: + # NOTE: Modeled after _Final's implementation when _FinalTypingBase available + class _Unpack(typing._FinalTypingBase, _root=True): + """A special typing construct to unpack a variadic type. For example: + + Shape = TypeVarTuple('Shape') + Batch = NewType('Batch', int) + + def add_batch_axis( + x: Array[Unpack[Shape]] + ) -> Array[Batch, Unpack[Shape]]: ... + + """ + __slots__ = ('__type__',) + __class__ = typing.TypeVar + + def __init__(self, tp=None, **kwds): + self.__type__ = tp + + def __getitem__(self, item): + cls = type(self) + if self.__type__ is None: + return cls(typing._type_check(item, + 'Unpack accepts only single type.'), + _root=True) + raise TypeError('Unpack cannot be further subscripted') + + def _eval_type(self, globalns, localns): + new_tp = typing._eval_type(self.__type__, globalns, localns) + if new_tp == self.__type__: + return self + return type(self)(new_tp, _root=True) + + def __repr__(self): + r = super().__repr__() + if self.__type__ is not None: + r += '[{}]'.format(typing._type_repr(self.__type__)) + return r + + def __hash__(self): + return hash((type(self).__name__, self.__type__)) + + def __eq__(self, other): + if not isinstance(other, _Unpack): + return NotImplemented + if self.__type__ is not None: + return self.__type__ == other.__type__ + return self is other + + # For 3.6 only + def _get_type_vars(self, tvars): + self.__type__._get_type_vars(tvars) + + Unpack = _Unpack(_root=True) + + def _is_unpack(obj): + return isinstance(obj, _Unpack) + + +class TypeVarTuple: + """Type variable tuple. + + Usage:: + + Ts = TypeVarTuple('Ts') + + In the same way that a normal type variable is a stand-in for a single + type such as ``int``, a type variable *tuple* is a stand-in for a *tuple* type such as + ``Tuple[int, str]``. + + Type variable tuples can be used in ``Generic`` declarations. + Consider the following example:: + + class Array(Generic[*Ts]): ... + + The ``Ts`` type variable tuple here behaves like ``tuple[T1, T2]``, + where ``T1`` and ``T2`` are type variables. To use these type variables + as type parameters of ``Array``, we must *unpack* the type variable tuple using + the star operator: ``*Ts``. The signature of ``Array`` then behaves + as if we had simply written ``class Array(Generic[T1, T2]): ...``. + In contrast to ``Generic[T1, T2]``, however, ``Generic[*Shape]`` allows + us to parameterise the class with an *arbitrary* number of type parameters. + + Type variable tuples can be used anywhere a normal ``TypeVar`` can. + This includes class definitions, as shown above, as well as function + signatures and variable annotations:: + + class Array(Generic[*Ts]): + + def __init__(self, shape: Tuple[*Ts]): + self._shape: Tuple[*Ts] = shape + + def get_shape(self) -> Tuple[*Ts]: + return self._shape + + shape = (Height(480), Width(640)) + x: Array[Height, Width] = Array(shape) + y = abs(x) # Inferred type is Array[Height, Width] + z = x + x # ... is Array[Height, Width] + x.get_shape() # ... is tuple[Height, Width] + + """ + + # Trick Generic __parameters__. + __class__ = typing.TypeVar + + def __iter__(self): + yield self.__unpacked__ + + def __init__(self, name): + self.__name__ = name + + # for pickling: + try: + def_mod = sys._getframe(1).f_globals.get('__name__', '__main__') + except (AttributeError, ValueError): + def_mod = None + if def_mod != 'typing_extensions': + self.__module__ = def_mod + + self.__unpacked__ = Unpack[self] + + def __repr__(self): + return self.__name__ + + def __hash__(self): + return object.__hash__(self) + + def __eq__(self, other): + return self is other + + def __reduce__(self): + return self.__name__ + + def __init_subclass__(self, *args, **kwds): + if '_root' not in kwds: + raise TypeError("Cannot subclass special typing classes") + + if not PEP_560: + # Only needed in 3.6. + def _get_type_vars(self, tvars): + if self not in tvars: + tvars.append(self) + + +if hasattr(typing, "reveal_type"): + reveal_type = typing.reveal_type +else: + def reveal_type(__obj: T) -> T: + """Reveal the inferred type of a variable. + + When a static type checker encounters a call to ``reveal_type()``, + it will emit the inferred type of the argument:: + + x: int = 1 + reveal_type(x) + + Running a static type checker (e.g., ``mypy``) on this example + will produce output similar to 'Revealed type is "builtins.int"'. + + At runtime, the function prints the runtime type of the + argument and returns it unchanged. + + """ + print(f"Runtime type is {type(__obj).__name__!r}", file=sys.stderr) + return __obj + + +if hasattr(typing, "assert_never"): + assert_never = typing.assert_never +else: + def assert_never(__arg: Never) -> Never: + """Assert to the type checker that a line of code is unreachable. + + Example:: + + def int_or_str(arg: int | str) -> None: + match arg: + case int(): + print("It's an int") + case str(): + print("It's a str") + case _: + assert_never(arg) + + If a type checker finds that a call to assert_never() is + reachable, it will emit an error. + + At runtime, this throws an exception when called. + + """ + raise AssertionError("Expected code to be unreachable") + + +if hasattr(typing, 'dataclass_transform'): + dataclass_transform = typing.dataclass_transform +else: + def dataclass_transform( + *, + eq_default: bool = True, + order_default: bool = False, + kw_only_default: bool = False, + field_descriptors: typing.Tuple[ + typing.Union[typing.Type[typing.Any], typing.Callable[..., typing.Any]], + ... + ] = (), + ) -> typing.Callable[[T], T]: + """Decorator that marks a function, class, or metaclass as providing + dataclass-like behavior. + + Example: + + from metaflow._vendor.v3_6.typing_extensions import dataclass_transform + + _T = TypeVar("_T") + + # Used on a decorator function + @dataclass_transform() + def create_model(cls: type[_T]) -> type[_T]: + ... + return cls + + @create_model + class CustomerModel: + id: int + name: str + + # Used on a base class + @dataclass_transform() + class ModelBase: ... + + class CustomerModel(ModelBase): + id: int + name: str + + # Used on a metaclass + @dataclass_transform() + class ModelMeta(type): ... + + class ModelBase(metaclass=ModelMeta): ... + + class CustomerModel(ModelBase): + id: int + name: str + + Each of the ``CustomerModel`` classes defined in this example will now + behave similarly to a dataclass created with the ``@dataclasses.dataclass`` + decorator. For example, the type checker will synthesize an ``__init__`` + method. + + The arguments to this decorator can be used to customize this behavior: + - ``eq_default`` indicates whether the ``eq`` parameter is assumed to be + True or False if it is omitted by the caller. + - ``order_default`` indicates whether the ``order`` parameter is + assumed to be True or False if it is omitted by the caller. + - ``kw_only_default`` indicates whether the ``kw_only`` parameter is + assumed to be True or False if it is omitted by the caller. + - ``field_descriptors`` specifies a static list of supported classes + or functions, that describe fields, similar to ``dataclasses.field()``. + + At runtime, this decorator records its arguments in the + ``__dataclass_transform__`` attribute on the decorated object. + + See PEP 681 for details. + + """ + def decorator(cls_or_fn): + cls_or_fn.__dataclass_transform__ = { + "eq_default": eq_default, + "order_default": order_default, + "kw_only_default": kw_only_default, + "field_descriptors": field_descriptors, + } + return cls_or_fn + return decorator + + +# We have to do some monkey patching to deal with the dual nature of +# Unpack/TypeVarTuple: +# - We want Unpack to be a kind of TypeVar so it gets accepted in +# Generic[Unpack[Ts]] +# - We want it to *not* be treated as a TypeVar for the purposes of +# counting generic parameters, so that when we subscript a generic, +# the runtime doesn't try to substitute the Unpack with the subscripted type. +if not hasattr(typing, "TypeVarTuple"): + typing._collect_type_vars = _collect_type_vars + typing._check_generic = _check_generic diff --git a/metaflow/_vendor/v3_6/zipp.LICENSE b/metaflow/_vendor/v3_6/zipp.LICENSE new file mode 100644 index 00000000000..353924be0e5 --- /dev/null +++ b/metaflow/_vendor/v3_6/zipp.LICENSE @@ -0,0 +1,19 @@ +Copyright Jason R. Coombs + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to +deal in the Software without restriction, including without limitation the +rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +sell copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. diff --git a/metaflow/_vendor/v3_6/zipp.py b/metaflow/_vendor/v3_6/zipp.py new file mode 100644 index 00000000000..26b723c1fd3 --- /dev/null +++ b/metaflow/_vendor/v3_6/zipp.py @@ -0,0 +1,329 @@ +import io +import posixpath +import zipfile +import itertools +import contextlib +import sys +import pathlib + +if sys.version_info < (3, 7): + from collections import OrderedDict +else: + OrderedDict = dict + + +__all__ = ['Path'] + + +def _parents(path): + """ + Given a path with elements separated by + posixpath.sep, generate all parents of that path. + + >>> list(_parents('b/d')) + ['b'] + >>> list(_parents('/b/d/')) + ['/b'] + >>> list(_parents('b/d/f/')) + ['b/d', 'b'] + >>> list(_parents('b')) + [] + >>> list(_parents('')) + [] + """ + return itertools.islice(_ancestry(path), 1, None) + + +def _ancestry(path): + """ + Given a path with elements separated by + posixpath.sep, generate all elements of that path + + >>> list(_ancestry('b/d')) + ['b/d', 'b'] + >>> list(_ancestry('/b/d/')) + ['/b/d', '/b'] + >>> list(_ancestry('b/d/f/')) + ['b/d/f', 'b/d', 'b'] + >>> list(_ancestry('b')) + ['b'] + >>> list(_ancestry('')) + [] + """ + path = path.rstrip(posixpath.sep) + while path and path != posixpath.sep: + yield path + path, tail = posixpath.split(path) + + +_dedupe = OrderedDict.fromkeys +"""Deduplicate an iterable in original order""" + + +def _difference(minuend, subtrahend): + """ + Return items in minuend not in subtrahend, retaining order + with O(1) lookup. + """ + return itertools.filterfalse(set(subtrahend).__contains__, minuend) + + +class CompleteDirs(zipfile.ZipFile): + """ + A ZipFile subclass that ensures that implied directories + are always included in the namelist. + """ + + @staticmethod + def _implied_dirs(names): + parents = itertools.chain.from_iterable(map(_parents, names)) + as_dirs = (p + posixpath.sep for p in parents) + return _dedupe(_difference(as_dirs, names)) + + def namelist(self): + names = super(CompleteDirs, self).namelist() + return names + list(self._implied_dirs(names)) + + def _name_set(self): + return set(self.namelist()) + + def resolve_dir(self, name): + """ + If the name represents a directory, return that name + as a directory (with the trailing slash). + """ + names = self._name_set() + dirname = name + '/' + dir_match = name not in names and dirname in names + return dirname if dir_match else name + + @classmethod + def make(cls, source): + """ + Given a source (filename or zipfile), return an + appropriate CompleteDirs subclass. + """ + if isinstance(source, CompleteDirs): + return source + + if not isinstance(source, zipfile.ZipFile): + return cls(_pathlib_compat(source)) + + # Only allow for FastLookup when supplied zipfile is read-only + if 'r' not in source.mode: + cls = CompleteDirs + + source.__class__ = cls + return source + + +class FastLookup(CompleteDirs): + """ + ZipFile subclass to ensure implicit + dirs exist and are resolved rapidly. + """ + + def namelist(self): + with contextlib.suppress(AttributeError): + return self.__names + self.__names = super(FastLookup, self).namelist() + return self.__names + + def _name_set(self): + with contextlib.suppress(AttributeError): + return self.__lookup + self.__lookup = super(FastLookup, self)._name_set() + return self.__lookup + + +def _pathlib_compat(path): + """ + For path-like objects, convert to a filename for compatibility + on Python 3.6.1 and earlier. + """ + try: + return path.__fspath__() + except AttributeError: + return str(path) + + +class Path: + """ + A pathlib-compatible interface for zip files. + + Consider a zip file with this structure:: + + . + ├── a.txt + └── b + ├── c.txt + └── d + └── e.txt + + >>> data = io.BytesIO() + >>> zf = zipfile.ZipFile(data, 'w') + >>> zf.writestr('a.txt', 'content of a') + >>> zf.writestr('b/c.txt', 'content of c') + >>> zf.writestr('b/d/e.txt', 'content of e') + >>> zf.filename = 'mem/abcde.zip' + + Path accepts the zipfile object itself or a filename + + >>> root = Path(zf) + + From there, several path operations are available. + + Directory iteration (including the zip file itself): + + >>> a, b = root.iterdir() + >>> a + Path('mem/abcde.zip', 'a.txt') + >>> b + Path('mem/abcde.zip', 'b/') + + name property: + + >>> b.name + 'b' + + join with divide operator: + + >>> c = b / 'c.txt' + >>> c + Path('mem/abcde.zip', 'b/c.txt') + >>> c.name + 'c.txt' + + Read text: + + >>> c.read_text() + 'content of c' + + existence: + + >>> c.exists() + True + >>> (b / 'missing.txt').exists() + False + + Coercion to string: + + >>> import os + >>> str(c).replace(os.sep, posixpath.sep) + 'mem/abcde.zip/b/c.txt' + + At the root, ``name``, ``filename``, and ``parent`` + resolve to the zipfile. Note these attributes are not + valid and will raise a ``ValueError`` if the zipfile + has no filename. + + >>> root.name + 'abcde.zip' + >>> str(root.filename).replace(os.sep, posixpath.sep) + 'mem/abcde.zip' + >>> str(root.parent) + 'mem' + """ + + __repr = "{self.__class__.__name__}({self.root.filename!r}, {self.at!r})" + + def __init__(self, root, at=""): + """ + Construct a Path from a ZipFile or filename. + + Note: When the source is an existing ZipFile object, + its type (__class__) will be mutated to a + specialized type. If the caller wishes to retain the + original type, the caller should either create a + separate ZipFile object or pass a filename. + """ + self.root = FastLookup.make(root) + self.at = at + + def open(self, mode='r', *args, pwd=None, **kwargs): + """ + Open this entry as text or binary following the semantics + of ``pathlib.Path.open()`` by passing arguments through + to io.TextIOWrapper(). + """ + if self.is_dir(): + raise IsADirectoryError(self) + zip_mode = mode[0] + if not self.exists() and zip_mode == 'r': + raise FileNotFoundError(self) + stream = self.root.open(self.at, zip_mode, pwd=pwd) + if 'b' in mode: + if args or kwargs: + raise ValueError("encoding args invalid for binary operation") + return stream + return io.TextIOWrapper(stream, *args, **kwargs) + + @property + def name(self): + return pathlib.Path(self.at).name or self.filename.name + + @property + def suffix(self): + return pathlib.Path(self.at).suffix or self.filename.suffix + + @property + def suffixes(self): + return pathlib.Path(self.at).suffixes or self.filename.suffixes + + @property + def stem(self): + return pathlib.Path(self.at).stem or self.filename.stem + + @property + def filename(self): + return pathlib.Path(self.root.filename).joinpath(self.at) + + def read_text(self, *args, **kwargs): + with self.open('r', *args, **kwargs) as strm: + return strm.read() + + def read_bytes(self): + with self.open('rb') as strm: + return strm.read() + + def _is_child(self, path): + return posixpath.dirname(path.at.rstrip("/")) == self.at.rstrip("/") + + def _next(self, at): + return self.__class__(self.root, at) + + def is_dir(self): + return not self.at or self.at.endswith("/") + + def is_file(self): + return self.exists() and not self.is_dir() + + def exists(self): + return self.at in self.root._name_set() + + def iterdir(self): + if not self.is_dir(): + raise ValueError("Can't listdir a file") + subs = map(self._next, self.root.namelist()) + return filter(self._is_child, subs) + + def __str__(self): + return posixpath.join(self.root.filename, self.at) + + def __repr__(self): + return self.__repr.format(self=self) + + def joinpath(self, *other): + next = posixpath.join(self.at, *map(_pathlib_compat, other)) + return self._next(self.root.resolve_dir(next)) + + __truediv__ = joinpath + + @property + def parent(self): + if not self.at: + return self.filename.parent + parent_at = posixpath.dirname(self.at.rstrip('/')) + if parent_at: + parent_at += '/' + return self._next(parent_at) diff --git a/metaflow/_vendor/vendor_any.txt b/metaflow/_vendor/vendor_any.txt new file mode 100644 index 00000000000..f7fc59bf3bb --- /dev/null +++ b/metaflow/_vendor/vendor_any.txt @@ -0,0 +1 @@ +click==7.1.2 diff --git a/metaflow/_vendor/vendor.txt b/metaflow/_vendor/vendor_v3_5.txt similarity index 66% rename from metaflow/_vendor/vendor.txt rename to metaflow/_vendor/vendor_v3_5.txt index 168fe394978..43295d035a6 100644 --- a/metaflow/_vendor/vendor.txt +++ b/metaflow/_vendor/vendor_v3_5.txt @@ -1,2 +1 @@ -click==7.1.2 importlib_metadata==2.1.3 diff --git a/metaflow/_vendor/vendor_v3_6.txt b/metaflow/_vendor/vendor_v3_6.txt new file mode 100644 index 00000000000..7ab36b23bb5 --- /dev/null +++ b/metaflow/_vendor/vendor_v3_6.txt @@ -0,0 +1 @@ +importlib_metadata==4.8.3 \ No newline at end of file diff --git a/metaflow/cli.py b/metaflow/cli.py index 2674757aa82..1bc1ba711a0 100644 --- a/metaflow/cli.py +++ b/metaflow/cli.py @@ -1,5 +1,4 @@ import inspect -import os import sys import traceback from datetime import datetime @@ -15,6 +14,7 @@ from . import namespace from . import current from .cli_args import cli_args +from .tagging_util import validate_tags from .util import ( resolve_identity, decompress_list, @@ -24,11 +24,12 @@ from .task import MetaflowTask from .exception import CommandException, MetaflowException from .graph import FlowGraph -from .datastore import DATASTORES, FlowDataStore, TaskDataStoreSet, TaskDataStore +from .datastore import FlowDataStore, TaskDataStoreSet, TaskDataStore from .runtime import NativeRuntime from .package import MetaflowPackage from .plugins import ( + DATASTORES, ENVIRONMENTS, LOGGING_SIDECARS, METADATA_PROVIDERS, @@ -44,8 +45,6 @@ ) from .metaflow_environment import MetaflowEnvironment from .pylint_wrapper import PyLint -from .event_logger import EventLogger -from .monitor import Monitor from .R import use_r, metaflow_r_version from .mflog import mflog, LOG_SOURCES from .unbounded_foreach import UBF_CONTROL, UBF_TASK @@ -111,7 +110,7 @@ def echo_always(line, **kwargs): click.secho(ERASE_TO_EOL, **kwargs) -def logger(body="", system_msg=False, head="", bad=False, timestamp=True): +def logger(body="", system_msg=False, head="", bad=False, timestamp=True, nl=True): if timestamp: if timestamp is True: dt = datetime.now() @@ -121,7 +120,7 @@ def logger(body="", system_msg=False, head="", bad=False, timestamp=True): click.secho(tstamp + " ", fg=LOGGER_TIMESTAMP, nl=False) if head: click.secho(head, fg=LOGGER_COLOR, nl=False) - click.secho(body, bold=system_msg, fg=LOGGER_BAD_COLOR if bad else None) + click.secho(body, bold=system_msg, fg=LOGGER_BAD_COLOR if bad else None, nl=nl) @click.group() @@ -174,10 +173,22 @@ def help(ctx): @cli.command(help="Output internal state of the flow graph.") +@click.option("--json", is_flag=True, help="Output the flow graph in JSON format.") @click.pass_obj -def output_raw(obj): - echo("Internal representation of the flow:", fg="magenta", bold=False) - echo_always(str(obj.graph), err=False) +def output_raw(obj, json): + if json: + import json as _json + + _msg = "Internal representation of the flow in JSON format:" + _graph_dict, _graph_struct = obj.graph.output_steps() + _graph = _json.dumps( + dict(graph=_graph_dict, graph_structure=_graph_struct), indent=4 + ) + else: + _graph = str(obj.graph) + _msg = "Internal representation of the flow:" + echo(_msg, fg="magenta", bold=False) + echo_always(_graph, err=False) @cli.command(help="Visualize the flow with Graphviz.") @@ -240,7 +251,7 @@ def dump(obj, input_path, private=None, max_value_size=None, include=None, file= run_id, step_name, task_id = parts else: raise CommandException( - "input_path should either be run_id/step_name" "or run_id/step_name/task_id" + "input_path should either be run_id/step_name or run_id/step_name/task_id" ) datastore_set = TaskDataStoreSet( @@ -386,7 +397,7 @@ def echo_unicode(line, **kwargs): log = ds.load_log_legacy(stream) if log and timestamps: raise CommandException( - "We can't show --timestamps for " "old runs. Sorry!" + "We can't show --timestamps for old runs. Sorry!" ) echo_unicode(log, nl=False) else: @@ -416,7 +427,13 @@ def echo_unicode(line, **kwargs): ) @click.option( "--input-paths", - help="A comma-separated list of pathspecs " "specifying inputs for this step.", + help="A comma-separated list of pathspecs specifying inputs for this step.", +) +@click.option( + "--input-paths-filename", + type=click.Path(exists=True, readable=True, dir_okay=False, resolve_path=True), + help="A filename containing the argument typically passed to `input-paths`", + hidden=True, ) @click.option( "--split-index", @@ -438,7 +455,7 @@ def echo_unicode(line, **kwargs): "--namespace", "opt_namespace", default=None, - help="Change namespace from the default (your username) to " "the specified tag.", + help="Change namespace from the default (your username) to the specified tag.", ) @click.option( "--retry-count", @@ -456,10 +473,17 @@ def echo_unicode(line, **kwargs): help="Pathspec of the origin task for this task to clone. Do " "not execute anything.", ) +@click.option( + "--clone-wait-only/--no-clone-wait-only", + default=False, + show_default=True, + help="If specified, waits for an external process to clone the task", + hidden=True, +) @click.option( "--clone-run-id", default=None, - help="Run id of the origin flow, if this task is part of a flow " "being resumed.", + help="Run id of the origin flow, if this task is part of a flow being resumed.", ) @click.option( "--with", @@ -473,7 +497,7 @@ def echo_unicode(line, **kwargs): "--ubf-context", default="none", type=click.Choice(["none", UBF_CONTROL, UBF_TASK]), - help="Provides additional context if this task is of type " "unbounded foreach.", + help="Provides additional context if this task is of type unbounded foreach.", ) @click.option( "--num-parallel", @@ -489,11 +513,13 @@ def step( run_id=None, task_id=None, input_paths=None, + input_paths_filename=None, split_index=None, opt_namespace=None, retry_count=None, max_user_code_retries=None, clone_only=None, + clone_wait_only=False, clone_run_id=None, decospecs=None, ubf_context="none", @@ -526,6 +552,10 @@ def step( cli_args._set_step_kwargs(step_kwargs) ctx.obj.metadata.add_sticky_tags(tags=opt_tag) + if not input_paths and input_paths_filename: + with open(input_paths_filename, mode="r", encoding="utf-8") as f: + input_paths = f.read().strip(" \n\"'") + paths = decompress_list(input_paths) if input_paths else [] task = MetaflowTask( @@ -539,7 +569,14 @@ def step( ubf_context, ) if clone_only: - task.clone_only(step_name, run_id, task_id, clone_only, retry_count) + task.clone_only( + step_name, + run_id, + task_id, + clone_only, + retry_count, + wait_only=clone_wait_only, + ) else: task.run_step( step_name, @@ -662,6 +699,26 @@ def wrapper(*args, **kwargs): help="ID of the run that should be resumed. By default, the " "last run executed locally.", ) +@click.option( + "--run-id", + default=None, + help="Run ID for the new run. By default, a new run-id will be generated", + hidden=True, +) +@click.option( + "--clone-only/--no-clone-only", + default=False, + show_default=True, + help="Only clone tasks without continuing execution", + hidden=True, +) +@click.option( + "--reentrant/--no-reentrant", + default=False, + show_default=True, + hidden=True, + help="If specified, allows this call to be called in parallel", +) @click.argument("step-to-rerun", required=False) @cli.command(help="Resume execution of a previous run of this flow.") @common_run_options @@ -671,6 +728,9 @@ def resume( tags=None, step_to_rerun=None, origin_run_id=None, + run_id=None, + clone_only=False, + reentrant=False, max_workers=None, max_num_splits=None, max_log_size=None, @@ -700,6 +760,17 @@ def resume( ) clone_steps = {step_to_rerun} + if run_id: + # Run-ids that are provided by the metadata service are always integers. + # External providers or run-ids (like external schedulers) always need to + # be non-integers to avoid any clashes. This condition ensures this. + try: + int(run_id) + except: + pass + else: + raise CommandException("run-id %s cannot be an integer" % run_id) + runtime = NativeRuntime( obj.flow, obj.graph, @@ -711,17 +782,19 @@ def resume( obj.entrypoint, obj.event_logger, obj.monitor, + run_id=run_id, clone_run_id=origin_run_id, + clone_only=clone_only, + reentrant=reentrant, clone_steps=clone_steps, max_workers=max_workers, max_num_splits=max_num_splits, max_log_size=max_log_size * 1024 * 1024, ) + write_run_id(run_id_file, runtime.run_id) runtime.persist_constants() runtime.execute() - write_run_id(run_id_file, runtime.run_id) - @parameters.add_custom_parameters(deploy_mode=True) @cli.command(help="Run the workflow locally.") @@ -784,6 +857,8 @@ def write_run_id(run_id_file, run_id): def before_run(obj, tags, decospecs): + validate_tags(tags) + # There's a --with option both at the top-level and for the run # subcommand. Why? # @@ -791,8 +866,8 @@ def before_run(obj, tags, decospecs): # This is a very common use case of --with. # # A downside is that we need to have the following decorators handling - # in two places in this module and we need to make sure that - # _init_step_decorators doesn't get called twice. + # in two places in this module and make sure _init_step_decorators + # doesn't get called twice. if decospecs: decorators._attach_decorators(obj.flow, decospecs) obj.graph = FlowGraph(obj.flow.__class__) @@ -802,6 +877,7 @@ def before_run(obj, tags, decospecs): decorators._init_step_decorators( obj.flow, obj.graph, obj.environment, obj.flow_datastore, obj.logger ) + obj.metadata.add_sticky_tags(tags=tags) # Package working directory only once per run. @@ -848,13 +924,13 @@ def version(obj): "--datastore", default=DEFAULT_DATASTORE, show_default=True, - type=click.Choice(DATASTORES), + type=click.Choice([d.TYPE for d in DATASTORES]), help="Data backend type", ) @click.option("--datastore-root", help="Root path for datastore") @click.option( "--package-suffixes", - help="A comma-separated list of file suffixes to include " "in the code package.", + help="A comma-separated list of file suffixes to include in the code package.", default=DEFAULT_PACKAGE_SUFFIXES, show_default=True, ) @@ -918,6 +994,7 @@ def start( cli_args._set_top_kwargs(ctx.params) ctx.obj.echo = echo ctx.obj.echo_always = echo_always + ctx.obj.is_quiet = quiet ctx.obj.graph = FlowGraph(ctx.obj.flow.__class__) ctx.obj.logger = logger ctx.obj.check = _check @@ -926,21 +1003,26 @@ def start( ctx.obj.package_suffixes = package_suffixes.split(",") ctx.obj.reconstruct_cli = _reconstruct_cli - ctx.obj.event_logger = EventLogger(event_logger) - ctx.obj.environment = [ e for e in ENVIRONMENTS + [MetaflowEnvironment] if e.TYPE == environment ][0](ctx.obj.flow) - ctx.obj.environment.validate_environment(echo) + ctx.obj.environment.validate_environment(echo, datastore) + + ctx.obj.event_logger = LOGGING_SIDECARS[event_logger]( + flow=ctx.obj.flow, env=ctx.obj.environment + ) + ctx.obj.event_logger.start() - ctx.obj.monitor = Monitor(monitor, ctx.obj.environment, ctx.obj.flow.name) + ctx.obj.monitor = MONITOR_SIDECARS[monitor]( + flow=ctx.obj.flow, env=ctx.obj.environment + ) ctx.obj.monitor.start() ctx.obj.metadata = [m for m in METADATA_PROVIDERS if m.TYPE == metadata][0]( ctx.obj.environment, ctx.obj.flow, ctx.obj.event_logger, ctx.obj.monitor ) - ctx.obj.datastore_impl = DATASTORES[datastore] + ctx.obj.datastore_impl = [d for d in DATASTORES if d.TYPE == datastore][0] if datastore_root is None: datastore_root = ctx.obj.datastore_impl.get_datastore_root_from_config( @@ -964,7 +1046,7 @@ def start( ) # It is important to initialize flow decorators early as some of the - # things they provide may be used by some of the objects initialize after. + # things they provide may be used by some of the objects initialized after. decorators._init_flow_decorators( ctx.obj.flow, ctx.obj.graph, @@ -1105,3 +1187,8 @@ def main(flow, args=None, handle_exceptions=True, entrypoint=None): sys.exit(1) else: raise + finally: + if hasattr(state, "monitor") and state.monitor is not None: + state.monitor.terminate() + if hasattr(state, "event_logger") and state.event_logger is not None: + state.event_logger.terminate() diff --git a/metaflow/cli_args.py b/metaflow/cli_args.py index 680fd1bdb3d..40918f984ff 100644 --- a/metaflow/cli_args.py +++ b/metaflow/cli_args.py @@ -57,7 +57,7 @@ def _options(mapping): for k, v in mapping.items(): # None or False arguments are ignored - # v needs to be explicitly False, not falsy, eg. 0 is an acceptable value + # v needs to be explicitly False, not falsy, e.g. 0 is an acceptable value if v is None or v is False: continue diff --git a/metaflow/client/core.py b/metaflow/client/core.py index 62ae7553115..bd268d0dc60 100644 --- a/metaflow/client/core.py +++ b/metaflow/client/core.py @@ -8,16 +8,17 @@ from itertools import chain from metaflow.metaflow_environment import MetaflowEnvironment +from metaflow.current import current from metaflow.exception import ( MetaflowNotFound, MetaflowNamespaceMismatch, MetaflowInternalError, ) - +from metaflow.includefile import IncludedFile from metaflow.metaflow_config import DEFAULT_METADATA, MAX_ATTEMPTS from metaflow.plugins import ENVIRONMENTS, METADATA_PROVIDERS from metaflow.unbounded_foreach import CONTROL_TASK_TAG -from metaflow.util import cached_property, resolve_identity, to_unicode +from metaflow.util import cached_property, resolve_identity, to_unicode, is_stringish from .filecache import FileCache @@ -47,6 +48,11 @@ def metadata(ms): for example, not allow access to information stored in remote metadata providers. + Note that you don't typically have to call this function directly. Usually + the metadata provider is set through the Metaflow configuration file. If you + need to switch between multiple providers, you can use the `METAFLOW_PROFILE` + environment variable to switch between configurations. + Parameters ---------- ms : string @@ -58,7 +64,7 @@ def metadata(ms): ------- string The description of the metadata selected (equivalent to the result of - get_metadata()) + get_metadata()). """ global current_metadata infos = ms.split("@", 1) @@ -90,11 +96,12 @@ def get_metadata(): """ Returns the current Metadata provider. - This call returns the current Metadata being used to return information - about Metaflow objects. + If this is not set explicitly using `metadata`, the default value is + determined through the Metaflow configuration. You can use this call to + check that your configuration is set up properly. - If this is not set explicitly using metadata(), the default value is - determined through environment variables. + If multiple configuration profiles are present, this call returns the one + selected through the `METAFLOW_PROFILE` environment variable. Returns ------- @@ -110,10 +117,8 @@ def get_metadata(): def default_metadata(): """ - Resets the Metadata provider to the default value. - - The default value of the Metadata provider is determined through a combination of - environment variables. + Resets the Metadata provider to the default value, that is, to the value + that was used prior to any `metadata` calls. Returns ------- @@ -121,6 +126,12 @@ def default_metadata(): The result of get_metadata() after resetting the provider. """ global current_metadata + + # We first check if we are in a flow -- if that is the case, we use the + # metadata provider that is being used there + if current._metadata_str: + return metadata(current._metadata_str) + default = [m for m in METADATA_PROVIDERS if m.TYPE == DEFAULT_METADATA] if default: current_metadata = default[0] @@ -174,15 +185,13 @@ def get_namespace(): def default_namespace(): """ - Sets or resets the namespace used to filter objects. - - The default namespace is in the form 'user:' and is intended to filter - objects belonging to the user. + Resets the namespace used to filter objects to the default one, i.e. the one that was + used prior to any `namespace` calls. Returns ------- string - The result of get_namespace() after + The result of get_namespace() after the namespace has been reset. """ global current_namespace current_namespace = resolve_identity() @@ -198,11 +207,10 @@ class Metaflow(object): Attributes ---------- - flows : List of all flows. - Returns the list of all flows. Note that only flows present in the set namespace will - be returned. A flow is present in a namespace if it has at least one run in the - namespace. - + flows : List[Flow] + Returns the list of all `Flow` objects known to this metadata provider. Note that only + flows present in the current namespace will be returned. A `Flow` is present in a namespace + if it has at least one run in the namespace. """ def __init__(self): @@ -302,7 +310,11 @@ class MetaflowObject(object): Attributes ---------- tags : Set - Tags associated with the object. + Tags associated with the run this object belongs to (user and system tags). + user_tags: Set + User tags associated with the run this object belongs to. + system_tags: Set + System tags associated with the run this object belongs to. created_at : datetime Date and time this object was first created. parent : MetaflowObject @@ -349,7 +361,7 @@ def __init__( raise MetaflowNotFound( "Attempt can only be smaller than %d" % MAX_ATTEMPTS ) - # NOTE: It is possible that no attempt exists but we can't + # NOTE: It is possible that no attempt exists, but we can't # distinguish between "attempt will happen" and "no such # attempt exists". @@ -379,6 +391,8 @@ def __init__( self._tags = frozenset( chain(self._object.get("system_tags") or [], self._object.get("tags") or []) ) + self._user_tags = frozenset(self._object.get("tags") or []) + self._system_tags = frozenset(self._object.get("system_tags") or []) if _namespace_check and not self.is_in_namespace(): raise MetaflowNamespaceMismatch(current_namespace) @@ -545,6 +559,30 @@ def tags(self): """ return self._tags + @property + def system_tags(self): + """ + System defined tags associated with this object. + + Returns + ------- + Set[string] + System tags associated with the object + """ + return self._system_tags + + @property + def user_tags(self): + """ + User defined tags associated with this object. + + Returns + ------- + Set[string] + User tags associated with the object + """ + return self._user_tags + @property def created_at(self): """ @@ -613,7 +651,7 @@ def parent(self): if self._parent is None: pathspec = self.pathspec parent_pathspec = pathspec[: pathspec.rfind("/")] - # Only artifacts and tasks have attempts right now so we get the + # Only artifacts and tasks have attempts right now, so we get the # right parent if we are an artifact. attempt_to_pass = self._attempt if self._NAME == "artifact" else None # We can skip the namespace check because if self._NAME = 'run', @@ -664,6 +702,33 @@ def path_components(self): class MetaflowData(object): + """ + Container of data artifacts produced by a `Task`. This object is + instantiated through `Task.data`. + + `MetaflowData` allows results to be retrieved by their name + through a convenient dot notation: + + ```python + Task(...).data.my_object + ``` + + You can also test the existence of an object + + ```python + if 'my_object' in Task(...).data: + print('my_object found') + ``` + + Note that this container relies on the local cache to load all data + artifacts. If your `Task` contains a lot of data, a more efficient + approach is to load artifacts individually like so + + ``` + Task(...)['my_object'].data + ``` + """ + def __init__(self, artifacts): self._artifacts = dict((art.id, art) for art in artifacts) @@ -682,22 +747,28 @@ def __repr__(self): class MetaflowCode(object): """ - Describes the code that is occasionally stored with a run. + Snapshot of the code used to execute this `Run`. Instantiate the object through + `Run(...).code` (if all steps are executed remotely) or `Task(...).code` for an + individual task. The code package is the same for all steps of a `Run`. + + `MetaflowCode` includes a package of the user-defined `FlowSpec` class and supporting + files, as well as a snapshot of the Metaflow library itself. - A code package will contain the version of Metaflow that was used (all the files comprising - the Metaflow library) as well as selected files from the directory containing the Python - file of the FlowSpec. + Currently, `MetaflowCode` objects are stored only for `Run`s that have at least one `Step` + executing outside the user's local environment. + + The `TarFile` for the `Run` is given by `Run(...).code.tarball` Attributes ---------- path : string - Location (in the datastore provider) of the code package + Location (in the datastore provider) of the code package. info : Dict - Dictionary of information related to this code-package + Dictionary of information related to this code-package. flowspec : string - Source code of the file containing the FlowSpec in this code package + Source code of the file containing the `FlowSpec` in this code package. tarball : TarFile - Tar ball containing all the code + Python standard library `tarfile.TarFile` archive containing all the code. """ def __init__(self, flow_name, code_package): @@ -775,16 +846,19 @@ def __str__(self): class DataArtifact(MetaflowObject): """ - A single data artifact and associated metadata. + A single data artifact and associated metadata. Note that this object does + not contain other objects as it is the leaf object in the hierarchy. Attributes ---------- data : object - The unpickled representation of the data contained in this artifact + The data contained in this artifact, that is, the object produced during + execution of this run. sha : string - Encoding representing the unique identity of this artifact + A unique ID of this artifact. finished_at : datetime - Alias for created_at + Corresponds roughly to the `Task.finished_at` time of the parent `Task`. + An alias for `DataArtifact.created_at`. """ _NAME = "artifact" @@ -809,6 +883,7 @@ def data(self): if filecache is None: # TODO: Pass proper environment to properly extract artifacts filecache = FileCache() + # "create" the metadata information that the datastore needs # to access this object. # TODO: We can store more information in the metadata, particularly @@ -824,12 +899,15 @@ def data(self): }, } if location.startswith(":root:"): - return filecache.get_artifact(ds_type, location[6:], meta, *components) + obj = filecache.get_artifact(ds_type, location[6:], meta, *components) else: # Older artifacts have a location information which we can use. - return filecache.get_artifact_by_location( + obj = filecache.get_artifact_by_location( ds_type, location, meta, *components ) + if isinstance(obj, IncludedFile): + return obj.decode(self.id) + return obj @property def size(self): @@ -895,48 +973,53 @@ def finished_at(self): class Task(MetaflowObject): """ - A Task represents an execution of a step. + A `Task` represents an execution of a `Step`. - As such, it contains all data artifacts associated with that execution as - well as all metadata associated with the execution. + It contains all `DataArtifact` objects produced by the task as + well as metadata related to execution. - Note that you can also get information about a specific *attempt* of a - task. By default, the latest finished attempt is returned but you can + Note that the `@retry` decorator may cause multiple attempts of + the task to be present. Usually you want the latest attempt, which + is what instantiating a `Task` object returns by default. If + you need to e.g. retrieve logs from a failed attempt, you can explicitly get information about a specific attempt by using the following syntax when creating a task: - `Task('flow/run/step/task', attempt=)`. Note that you will not be able to - access a specific attempt of a task through the `.tasks` method of a step - for example (that will always return the latest attempt). + + `Task('flow/run/step/task', attempt=)` + + where `attempt=0` corresponds to the first attempt etc. Attributes ---------- metadata : List[Metadata] - List of all metadata associated with the task + List of all metadata events associated with the task. metadata_dict : Dict - Dictionary where the keys are the names of the metadata and the value are the values - associated with those names + A condensed version of `metadata`: A dictionary where keys + are names of metadata events and values the latest corresponding event. data : MetaflowData - Container of all data artifacts produced by this task + Container of all data artifacts produced by this task. Note that this + call downloads all data locally, so it can be slower than accessing + artifacts individually. See `MetaflowData` for more information. artifacts : MetaflowArtifacts - Container of DataArtifact objects produced by this task + Container of `DataArtifact` objects produced by this task. successful : boolean - True if the task successfully completed + True if the task completed successfully. finished : boolean - True if the task completed + True if the task completed. exception : object - Exception raised by this task if there was one + Exception raised by this task if there was one. finished_at : datetime - Time this task finished + Time this task finished. runtime_name : string - Runtime this task was executed on + Runtime this task was executed on. stdout : string - Standard output for the task execution + Standard output for the task execution. stderr : string - Standard error output for the task execution + Standard error output for the task execution. code : MetaflowCode - Code package for this task (if present) + Code package for this task (if present). See `MetaflowCode`. environment_info : Dict - Information about the execution environment (for example Conda) + Information about the execution environment. """ _NAME = "task" @@ -967,16 +1050,59 @@ def metadata(self): self._NAME, "metadata", None, self._attempt, *self.path_components ) all_metadata = all_metadata if all_metadata else [] - return [ - Metadata( - name=obj.get("field_name"), - value=obj.get("value"), - created_at=obj.get("ts_epoch"), - type=obj.get("type"), - task=self, + + # For "clones" (ie: they have an origin-run-id AND a origin-task-id), we + # copy a set of metadata from the original task. This is needed to make things + # like logs work (which rely on having proper values for ds-root for example) + origin_run_id = None + origin_task_id = None + result = [] + existing_keys = [] + for obj in all_metadata: + result.append( + Metadata( + name=obj.get("field_name"), + value=obj.get("value"), + created_at=obj.get("ts_epoch"), + type=obj.get("type"), + task=self, + ) ) - for obj in all_metadata - ] + existing_keys.append(obj.get("field_name")) + if obj.get("field_name") == "origin-run-id": + origin_run_id = obj.get("value") + elif obj.get("field_name") == "origin-task-id": + origin_task_id = obj.get("value") + + if origin_task_id: + # This is a "cloned" task. We consider that it has the same + # metadata as the last attempt of the cloned task. + + origin_obj_pathcomponents = self.path_components + origin_obj_pathcomponents[1] = origin_run_id + origin_obj_pathcomponents[3] = origin_task_id + origin_task = Task( + "/".join(origin_obj_pathcomponents), _namespace_check=False + ) + latest_metadata = { + m.name: m + for m in sorted(origin_task.metadata, key=lambda m: m.created_at) + } + # We point to ourselves in the Metadata object + for v in latest_metadata.values(): + if v.name in existing_keys: + continue + result.append( + Metadata( + name=v.name, + value=v.value, + created_at=v.created_at, + type=v.type, + task=self, + ) + ) + + return result @property def metadata_dict(self): @@ -1280,35 +1406,52 @@ def environment_info(self): if not env_type: return None env = [m for m in ENVIRONMENTS + [MetaflowEnvironment] if m.TYPE == env_type][0] - return env.get_client_info(self.path_components[0], self.metadata_dict) + meta_dict = self.metadata_dict + return env.get_client_info(self.path_components[0], meta_dict) def _load_log(self, stream): - log_location = self.metadata_dict.get("log_location_%s" % stream) + meta_dict = self.metadata_dict + log_location = meta_dict.get("log_location_%s" % stream) if log_location: return self._load_log_legacy(log_location, stream) else: - return "".join(line + "\n" for _, line in self.loglines(stream)) + return "".join( + line + "\n" for _, line in self.loglines(stream, meta_dict=meta_dict) + ) def _get_logsize(self, stream): - log_location = self.metadata_dict.get("log_location_%s" % stream) + meta_dict = self.metadata_dict + log_location = meta_dict.get("log_location_%s" % stream) if log_location: return self._legacy_log_size(log_location, stream) else: - return self._log_size(stream) + return self._log_size(stream, meta_dict) - def loglines(self, stream, as_unicode=True): + def loglines(self, stream, as_unicode=True, meta_dict=None): """ Return an iterator over (utc_timestamp, logline) tuples. - If as_unicode=False, logline is returned as a byte object. Otherwise, - it is returned as a (unicode) string. + Parameters + ---------- + stream : string + Either 'stdout' or 'stderr'. + as_unicode : boolean + If as_unicode=False, each logline is returned as a byte object. Otherwise, + it is returned as a (unicode) string. + + Returns + ------- + Iterator[(datetime, string)] + Iterator over timestamp, logline pairs. """ from metaflow.mflog.mflog import merge_logs global filecache - ds_type = self.metadata_dict.get("ds-type") - ds_root = self.metadata_dict.get("ds-root") + if meta_dict is None: + meta_dict = self.metadata_dict + ds_type = meta_dict.get("ds-type") + ds_root = meta_dict.get("ds-root") if ds_type is None or ds_root is None: yield None, "" return @@ -1355,11 +1498,11 @@ def _legacy_log_size(self, log_location, logtype): ds_type, location, logtype, int(attempt), *self.path_components ) - def _log_size(self, stream): + def _log_size(self, stream, meta_dict): global filecache - ds_type = self.metadata_dict.get("ds-type") - ds_root = self.metadata_dict.get("ds-root") + ds_type = meta_dict.get("ds-type") + ds_root = meta_dict.get("ds-root") if ds_type is None or ds_root is None: return 0 if filecache is None: @@ -1373,20 +1516,21 @@ def _log_size(self, stream): class Step(MetaflowObject): """ - A Step represents a user-defined Step (a method annotated with the @step decorator). + A `Step` represents a user-defined step, that is, a method annotated with the `@step` decorator. - As such, it contains all Tasks associated with the step (ie: all executions of the - Step). A linear Step will have only one associated task whereas a foreach Step will have - multiple Tasks. + It contains `Task` objects associated with the step, that is, all executions of the + `Step`. The step may contain multiple `Task`s in the case of a foreach step. Attributes ---------- task : Task - Returns a Task object from the step + The first `Task` object in this step. This is a shortcut for retrieving the only + task contained in a non-foreach step. finished_at : datetime - Time this step finished (time of completion of the last task) + Time when the latest `Task` of this step finished. Note that in the case of foreaches, + this time may change during execution of the step. environment_info : Dict - Information about the execution environment (for example Conda) + Information about the execution environment. """ _NAME = "step" @@ -1410,41 +1554,45 @@ def task(self): def tasks(self, *tags): """ - Returns an iterator over all the tasks in the step. + [Legacy function - do not use] - An optional filter is available that allows you to filter on tags. - If tags are specified, only tasks associated with all specified tags - are returned. + Returns an iterator over all `Task` objects in the step. This is an alias + to iterating the object itself, i.e. + ``` + list(Step(...)) == list(Step(...).tasks()) + ``` Parameters ---------- tags : string - Tags to match + No op (legacy functionality) Returns ------- Iterator[Task] - Iterator over Task objects in this step + Iterator over all `Task` objects in this step. """ return self._filtered_children(*tags) @property def control_task(self): """ + [Unpublished API - use with caution!] + Returns a Control Task object belonging to this step. This is useful when the step only contains one control task. + Returns ------- Task A control task in the step """ - children = super(Step, self).__iter__() - for t in children: - if CONTROL_TASK_TAG in t.tags: - return t + return next(self.control_tasks(), None) def control_tasks(self, *tags): """ + [Unpublished API - use with caution!] + Returns an iterator over all the control tasks in the step. An optional filter is available that allows you to filter on tags. The control tasks returned if the filter is specified will contain all the @@ -1459,11 +1607,23 @@ def control_tasks(self, *tags): Iterator over Control Task objects in this step """ children = super(Step, self).__iter__() - filter_tags = [CONTROL_TASK_TAG] - filter_tags.extend(tags) for child in children: - if all(tag in child.tags for tag in filter_tags): + # first filter by standard tag filters + if not all(tag in child.tags for tag in tags): + continue + # Then look for control task indicator in one of two ways + # Look in tags - this path will activate for metadata service + # backends that pre-date tag mutation release + if CONTROL_TASK_TAG in child.tags: yield child + else: + # Look in task metadata + for task_metadata in child.metadata: + if ( + task_metadata.name == "internal_task_type" + and task_metadata.value == CONTROL_TASK_TAG + ): + yield child def __iter__(self): children = super(Step, self).__iter__() @@ -1511,24 +1671,22 @@ def environment_info(self): class Run(MetaflowObject): """ - A Run represents an execution of a Flow - - As such, it contains all Steps associated with the flow. + A `Run` represents an execution of a `Flow`. It is a container of `Step`s. Attributes ---------- data : MetaflowData - Container of all data artifacts produced by this run + a shortcut to run['end'].task.data, i.e. data produced by this run. successful : boolean - True if the run successfully completed + True if the run completed successfully. finished : boolean - True if the run completed + True if the run completed. finished_at : datetime - Time this run finished + Time this run finished. code : MetaflowCode - Code package for this run (if present) + Code package for this run (if present). See `MetaflowCode`. end_task : Task - Task for the end step (if it is present already) + `Task` for the end step (if it is present already). """ _NAME = "run" @@ -1541,21 +1699,23 @@ def _iter_filter(self, x): def steps(self, *tags): """ - Returns an iterator over all the steps in the run. + [Legacy function - do not use] - An optional filter is available that allows you to filter on tags. - If tags are specified, only steps associated with all specified tags - are returned. + Returns an iterator over all `Step` objects in the step. This is an alias + to iterating the object itself, i.e. + ``` + list(Run(...)) == list(Run(...).steps()) + ``` Parameters ---------- tags : string - Tags to match + No op (legacy functionality) Returns ------- Iterator[Step] - Iterator over Step objects in this run + Iterator over `Step` objects in this run. """ return self._filtered_children(*tags) @@ -1667,20 +1827,134 @@ def end_task(self): return end_step.task + def add_tag(self, tag): + """ + Add a tag to this `Run`. + + Note that if the tag is already a system tag, it is not added as a user tag, + and no error is thrown. + + Parameters + ---------- + tag : string + Tag to add. + """ + + # For backwards compatibility with Netflix's early version of this functionality, + # this function shall accept both an individual tag AND iterables of tags. + # + # Iterable of tags support shall be removed in future once existing + # usage has been migrated off. + if is_stringish(tag): + tag = [tag] + return self.replace_tag([], tag) + + def add_tags(self, tags): + """ + Add one or more tags to this `Run`. + + Note that if any tag is already a system tag, it is not added as a user tag + and no error is thrown. + + Parameters + ---------- + tags : Iterable[string] + Tags to add. + """ + return self.replace_tag([], tags) + + def remove_tag(self, tag): + """ + Remove one tag from this `Run`. + + Removing a system tag is an error. Removing a non-existent + user tag is a no-op. + + Parameters + ---------- + tag : string + Tag to remove. + """ + + # For backwards compatibility with Netflix's early version of this functionality, + # this function shall accept both an individual tag AND iterables of tags. + # + # Iterable of tags support shall be removed in future once existing + # usage has been migrated off. + if is_stringish(tag): + tag = [tag] + return self.replace_tag(tag, []) + + def remove_tags(self, tags): + """ + Remove one or more tags to this `Run`. + + Removing a system tag will result in an error. Removing a non-existent + user tag is a no-op. + + Parameters + ---------- + tags : Iterable[string] + Tags to remove. + """ + return self.replace_tags(tags, []) + + def replace_tag(self, tag_to_remove, tag_to_add): + """ + Remove a tag and add a tag atomically. Removal is done first. + The rules for `Run.add_tag` and `Run.remove_tag` also apply here. + + Parameters + ---------- + tag_to_remove : string + Tag to remove. + tag_to_add : string + Tag to add. + """ + + # For backwards compatibility with Netflix's early version of this functionality, + # this function shall accept both individual tags AND iterables of tags. + # + # Iterable of tags support shall be removed in future once existing + # usage has been migrated off. + if is_stringish(tag_to_remove): + tag_to_remove = [tag_to_remove] + if is_stringish(tag_to_add): + tag_to_add = [tag_to_add] + return self.replace_tags(tag_to_remove, tag_to_add) + + def replace_tags(self, tags_to_remove, tags_to_add): + """ + Remove and add tags atomically; the removal is done first. + The rules for `Run.add_tag` and `Run.remove_tag` also apply here. + + Parameters + ---------- + tags_to_remove : Iterable[string] + Tags to remove. + tags_to_add : Iterable[string] + Tags to add. + """ + flow_id = self.path_components[0] + final_user_tags = self._metaflow.metadata.mutate_user_tags_for_run( + flow_id, self.id, tags_to_remove=tags_to_remove, tags_to_add=tags_to_add + ) + # refresh Run object with the latest tags + self._user_tags = frozenset(final_user_tags) + self._tags = frozenset([*self._user_tags, *self._system_tags]) + class Flow(MetaflowObject): """ A Flow represents all existing flows with a certain name, in other words, - classes derived from 'FlowSpec' - - As such, it contains all Runs (executions of a flow) related to this flow. + classes derived from `FlowSpec`. A container of `Run` objects. Attributes ---------- latest_run : Run - Latest Run (in progress or completed, successfully or not) of this Flow + Latest `Run` (in progress or completed, successfully or not) of this flow. latest_successful_run : Run - Latest successfully completed Run of this Flow + Latest successfully completed `Run` of this flow. """ _NAME = "flow" @@ -1722,21 +1996,21 @@ def latest_successful_run(self): def runs(self, *tags): """ - Returns an iterator over all the runs in the flow. + Returns an iterator over all `Run`s of this flow. An optional filter is available that allows you to filter on tags. - If tags are specified, only runs associated with all specified tags - are returned. + If multiple tags are specified, only runs that have all the + specified tags are returned. Parameters ---------- tags : string - Tags to match + Tags to match. Returns ------- Iterator[Run] - Iterator over Run objects in this flow + Iterator over `Run` objects in this flow. """ return self._filtered_children(*tags) diff --git a/metaflow/client/filecache.py b/metaflow/client/filecache.py index 5adf1a06c60..8c7d945d588 100644 --- a/metaflow/client/filecache.py +++ b/metaflow/client/filecache.py @@ -6,7 +6,7 @@ from tempfile import NamedTemporaryFile from hashlib import sha1 -from metaflow.datastore import DATASTORES, FlowDataStore +from metaflow.datastore import FlowDataStore from metaflow.datastore.content_addressed_store import BlobCache from metaflow.exception import MetaflowException from metaflow.metaflow_config import ( @@ -16,6 +16,8 @@ CLIENT_CACHE_MAX_TASKDATASTORE_COUNT, ) +from metaflow.plugins import DATASTORES + NEW_FILE_QUARANTINE = 10 if sys.version_info[0] >= 3 and sys.version_info[1] >= 2: @@ -23,7 +25,6 @@ def od_move_to_end(od, key): od.move_to_end(key) - else: # Not very efficient but works and most people are on 3.2+ def od_move_to_end(od, key): @@ -320,7 +321,7 @@ def _task_ds_id(ds_type, ds_root, flow_name, run_id, step_name, task_id, attempt def _garbage_collect(self): now = time.time() - while self._objects and self._total > self._max_size * 1024 ** 2: + while self._objects and self._total > self._max_size * 1024**2: if now - self._objects[0][0] < NEW_FILE_QUARANTINE: break ctime, size, path = self._objects.pop(0) @@ -345,10 +346,10 @@ def _makedirs(path): @staticmethod def _get_datastore_storage_impl(ds_type): - storage_impl = DATASTORES.get(ds_type, None) - if storage_impl is None: + storage_impl = [d for d in DATASTORES if d.TYPE == ds_type] + if len(storage_impl) == 0: raise FileCacheException("Datastore %s was not found" % ds_type) - return storage_impl + return storage_impl[0] def _get_flow_datastore(self, ds_type, ds_root, flow_name): cache_id = self._flow_ds_id(ds_type, ds_root, flow_name) diff --git a/metaflow/cmd/__init__.py b/metaflow/cmd/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/metaflow/main_cli.py b/metaflow/cmd/configure_cmd.py similarity index 62% rename from metaflow/main_cli.py rename to metaflow/cmd/configure_cmd.py index 709bee87ba1..c5457f2911b 100644 --- a/metaflow/main_cli.py +++ b/metaflow/cmd/configure_cmd.py @@ -1,270 +1,18 @@ -from metaflow._vendor import click import json import os -import shutil +import sys from os.path import expanduser -from metaflow.datastore.local_storage import LocalStorage -from metaflow.metaflow_config import DATASTORE_LOCAL_DIR +from metaflow.util import to_unicode +from metaflow._vendor import click from metaflow.util import to_unicode -def makedirs(path): - # This is for python2 compatibility. - # Python3 has os.makedirs(exist_ok=True). - try: - os.makedirs(path) - except OSError as x: - if x.errno == 17: - return - else: - raise - - -def echo_dev_null(*args, **kwargs): - pass - - -def echo_always(line, **kwargs): - click.secho(line, **kwargs) - - -@click.group(invoke_without_command=True) -@click.pass_context -def main(ctx): - global echo - echo = echo_always - - import metaflow - - echo("Metaflow ", fg="magenta", bold=True, nl=False) - - if ctx.invoked_subcommand is None: - echo("(%s): " % metaflow.__version__, fg="magenta", bold=False, nl=False) - else: - echo("(%s)\n" % metaflow.__version__, fg="magenta", bold=False) - - if ctx.invoked_subcommand is None: - echo("More data science, less engineering\n", fg="magenta") - - # metaflow URL - echo("http://docs.metaflow.org", fg="cyan", nl=False) - echo(" - Read the documentation") - - # metaflow chat - echo("http://chat.metaflow.org", fg="cyan", nl=False) - echo(" - Chat with us") - - # metaflow help email - echo("help@metaflow.org", fg="cyan", nl=False) - echo(" - Get help by email\n") - - # print a short list of next steps. - short_help = { - "tutorials": "Browse and access metaflow tutorials.", - "configure": "Configure metaflow to access the cloud.", - "status": "Display the current working tree.", - "help": "Show all available commands to run.", - } - - echo("Commands:", bold=False) - - for cmd, desc in short_help.items(): - echo(" metaflow {0:<10} ".format(cmd), fg="cyan", bold=False, nl=False) - - echo("%s" % desc) - - -@main.command(help="Show all available commands.") -@click.pass_context -def help(ctx): - print(ctx.parent.get_help()) - - -@main.command(help="Show flows accessible from the current working tree.") -def status(): - from metaflow.client import get_metadata - - res = get_metadata() - if res: - res = res.split("@") - else: - raise click.ClickException("Unknown status: cannot find a Metadata provider") - if res[0] == "service": - echo("Using Metadata provider at: ", nl=False) - echo('"%s"\n' % res[1], fg="cyan") - echo("To list available flows, type:\n") - echo("1. python") - echo("2. from metaflow import Metaflow") - echo("3. list(Metaflow())") - return - - from metaflow.client import namespace, metadata, Metaflow - - # Get the local data store path - path = LocalStorage.get_datastore_root_from_config(echo, create_on_absent=False) - # Throw an exception - if path is None: - raise click.ClickException( - "Could not find " - + click.style('"%s"' % DATASTORE_LOCAL_DIR, fg="red") - + " in the current working tree." - ) - - stripped_path = os.path.dirname(path) - namespace(None) - metadata("local@%s" % stripped_path) - echo("Working tree found at: ", nl=False) - echo('"%s"\n' % stripped_path, fg="cyan") - echo("Available flows:", fg="cyan", bold=True) - for flow in Metaflow(): - echo("* %s" % flow, fg="cyan") - - -@main.group(help="Browse and access the metaflow tutorial episodes.") -def tutorials(): - pass - - -def get_tutorials_dir(): - metaflow_dir = os.path.dirname(__file__) - package_dir = os.path.dirname(metaflow_dir) - tutorials_dir = os.path.join(package_dir, "metaflow", "tutorials") - - return tutorials_dir - - -def get_tutorial_metadata(tutorial_path): - metadata = {} - with open(os.path.join(tutorial_path, "README.md")) as readme: - content = readme.read() - - paragraphs = [paragraph.strip() for paragraph in content.split("#") if paragraph] - metadata["description"] = paragraphs[0].split("**")[1] - header = paragraphs[0].split("\n") - header = header[0].split(":") - metadata["episode"] = header[0].strip()[len("Episode ") :] - metadata["title"] = header[1].strip() - - for paragraph in paragraphs[1:]: - if paragraph.startswith("Before playing"): - lines = "\n".join(paragraph.split("\n")[1:]) - metadata["prereq"] = lines.replace("```", "") - - if paragraph.startswith("Showcasing"): - lines = "\n".join(paragraph.split("\n")[1:]) - metadata["showcase"] = lines.replace("```", "") - - if paragraph.startswith("To play"): - lines = "\n".join(paragraph.split("\n")[1:]) - metadata["play"] = lines.replace("```", "") - - return metadata - - -def get_all_episodes(): - episodes = [] - for name in sorted(os.listdir(get_tutorials_dir())): - # Skip hidden files (like .gitignore) - if not name.startswith("."): - episodes.append(name) - return episodes - - -@tutorials.command(help="List the available episodes.") -def list(): - echo("Episodes:", fg="cyan", bold=True) - for name in get_all_episodes(): - path = os.path.join(get_tutorials_dir(), name) - metadata = get_tutorial_metadata(path) - echo("* {0: <20} ".format(metadata["episode"]), fg="cyan", nl=False) - echo("- {0}".format(metadata["title"])) - - echo("\nTo pull the episodes, type: ") - echo("metaflow tutorials pull", fg="cyan") - - -def validate_episode(episode): - src_dir = os.path.join(get_tutorials_dir(), episode) - if not os.path.isdir(src_dir): - raise click.BadArgumentUsage( - "Episode " - + click.style('"{0}"'.format(episode), fg="red") - + " does not exist." - " To see a list of available episodes, " - "type:\n" + click.style("metaflow tutorials list", fg="cyan") - ) - - -def autocomplete_episodes(ctx, args, incomplete): - return [k for k in get_all_episodes() if incomplete in k] - +from .util import echo_always, makedirs -@tutorials.command(help="Pull episodes " "into your current working directory.") -@click.option( - "--episode", - default="", - help="Optional episode name " "to pull only a single episode.", -) -def pull(episode): - tutorials_dir = get_tutorials_dir() - if not episode: - episodes = get_all_episodes() - else: - episodes = [episode] - # Validate that the list is valid. - for episode in episodes: - validate_episode(episode) - # Create destination `metaflow-tutorials` dir. - dst_parent = os.path.join(os.getcwd(), "metaflow-tutorials") - makedirs(dst_parent) - - # Pull specified episodes. - for episode in episodes: - dst_dir = os.path.join(dst_parent, episode) - # Check if episode has already been pulled before. - if os.path.exists(dst_dir): - if click.confirm( - "Episode " - + click.style('"{0}"'.format(episode), fg="red") - + " has already been pulled before. Do you wish " - "to delete the existing version?" - ): - shutil.rmtree(dst_dir) - else: - continue - echo("Pulling episode ", nl=False) - echo('"{0}"'.format(episode), fg="cyan", nl=False) - # TODO: Is the following redundant? - echo(" into your current working directory.") - # Copy from (local) metaflow package dir to current. - src_dir = os.path.join(tutorials_dir, episode) - shutil.copytree(src_dir, dst_dir) - - echo("\nTo know more about an episode, type:\n", nl=False) - echo("metaflow tutorials info [EPISODE]", fg="cyan") - - -@tutorials.command(help="Find out more about an episode.") -@click.argument("episode", autocompletion=autocomplete_episodes) -def info(episode): - validate_episode(episode) - src_dir = os.path.join(get_tutorials_dir(), episode) - metadata = get_tutorial_metadata(src_dir) - echo("Synopsis:", fg="cyan", bold=True) - echo("%s" % metadata["description"]) - - echo("\nShowcasing:", fg="cyan", bold=True, nl=True) - echo("%s" % metadata["showcase"]) - - if "prereq" in metadata: - echo("\nBefore playing:", fg="cyan", bold=True, nl=True) - echo("%s" % metadata["prereq"]) - - echo("\nTo play:", fg="cyan", bold=True) - echo("%s" % metadata["play"]) +echo = echo_always # NOTE: This code needs to be in sync with metaflow/metaflow_config.py. METAFLOW_CONFIGURATION_DIR = expanduser( @@ -272,7 +20,12 @@ def info(episode): ) -@main.group(help="Configure Metaflow to access the cloud.") +@click.group() +def cli(): + pass + + +@cli.group(help="Configure Metaflow to access the cloud.") def configure(): makedirs(METAFLOW_CONFIGURATION_DIR) @@ -283,7 +36,7 @@ def get_config_path(profile): return path -def overwrite_config(profile): +def confirm_overwrite_config(profile): path = get_config_path(profile) if os.path.exists(path): if not click.confirm( @@ -424,7 +177,7 @@ def import_from(profile, input_filename): echo('"%s"' % input_path, fg="cyan") # Persist configuration. - overwrite_config(profile) + confirm_overwrite_config(profile) persist_env(env_dict, profile) @@ -436,8 +189,16 @@ def import_from(profile, input_filename): help="Configure a named profile. Activate the profile by setting " "`METAFLOW_PROFILE` environment variable.", ) -def sandbox(profile): - overwrite_config(profile) +@click.option( + "--overwrite/--no-overwrite", + "-o/", + default=False, + show_default=True, + help="Overwrite profile configuration without asking", +) +def sandbox(profile, overwrite): + if not overwrite: + confirm_overwrite_config(profile) # Prompt for user input. encoded_str = click.prompt( "Following instructions from " @@ -447,7 +208,8 @@ def sandbox(profile): ) # Decode the bytes to env_dict. try: - import base64, zlib + import base64 + import zlib from metaflow.util import to_bytes env_dict = json.loads( @@ -501,6 +263,44 @@ def configure_s3_datastore(existing_env): return env +def configure_azure_datastore(existing_env): + env = {} + # Set Azure Blob Storage as default datastore. + env["METAFLOW_DEFAULT_DATASTORE"] = "azure" + # Set Azure Blob Storage folder for datastore. + # TODO rename this Blob Endpoint! + env["METAFLOW_AZURE_STORAGE_BLOB_SERVICE_ENDPOINT"] = click.prompt( + cyan("[METAFLOW_AZURE_STORAGE_BLOB_SERVICE_ENDPOINT]") + + " Azure Storage Account URL, for the account holding the Blob container to be used. " + + "(E.g. https://.blob.core.windows.net/)", + default=existing_env.get("METAFLOW_AZURE_STORAGE_BLOB_SERVICE_ENDPOINT"), + show_default=True, + ) + env["METAFLOW_DATASTORE_SYSROOT_AZURE"] = click.prompt( + cyan("[METAFLOW_DATASTORE_SYSROOT_AZURE]") + + " Azure Blob Storage folder for Metaflow artifact storage " + + "(Format: /)", + default=existing_env.get("METAFLOW_DATASTORE_SYSROOT_AZURE"), + show_default=True, + ) + return env + + +def configure_gs_datastore(existing_env): + env = {} + # Set Google Cloud Storage as default datastore. + env["METAFLOW_DEFAULT_DATASTORE"] = "gs" + # Set Google Cloud Storage folder for datastore. + env["METAFLOW_DATASTORE_SYSROOT_GS"] = click.prompt( + cyan("[METAFLOW_DATASTORE_SYSROOT_GS]") + + " Google Cloud Storage folder for Metaflow artifact storage " + + "(Format: gs:///)", + default=existing_env.get("METAFLOW_DATASTORE_SYSROOT_GS"), + show_default=True, + ) + return env + + def configure_metadata_service(existing_env): empty_profile = False if not existing_env: @@ -520,7 +320,7 @@ def configure_metadata_service(existing_env): cyan("[METAFLOW_SERVICE_INTERNAL_URL]") + yellow(" (optional)") + " URL for Metaflow Service " - + "(Accessible only within VPC).", + + "(Accessible only within VPC [AWS] or a Kubernetes cluster [if the service runs in one]).", default=existing_env.get( "METAFLOW_SERVICE_INTERNAL_URL", env["METAFLOW_SERVICE_URL"] ), @@ -537,7 +337,83 @@ def configure_metadata_service(existing_env): return env -def configure_datastore_and_metadata(existing_env): +def configure_azure_datastore_and_metadata(existing_env): + empty_profile = False + if not existing_env: + empty_profile = True + env = {} + + # Configure Azure Blob Storage as the datastore. + use_azure_as_datastore = click.confirm( + "\nMetaflow can use " + + yellow("Azure Blob Storage as the storage backend") + + " for all code and data artifacts on " + + "Azure.\nAzure Blob Storage is a strict requirement if you " + + "intend to execute your flows on a Kubernetes cluster on Azure (AKS or self-managed)" + + ".\nWould you like to configure Azure Blob Storage " + + "as the default storage backend?", + default=empty_profile + or existing_env.get("METAFLOW_DEFAULT_DATASTORE", "") == "azure", + abort=False, + ) + if use_azure_as_datastore: + env.update(configure_azure_datastore(existing_env)) + + # Configure Metadata service for tracking. + if click.confirm( + "\nMetaflow can use a " + + yellow("remote Metadata Service to track") + + " and persist flow execution metadata.\nConfiguring the " + "service is a requirement if you intend to schedule your " + "flows with Kubernetes on Azure (AKS or self-managed).\nWould you like to " + "configure the Metadata Service?", + default=empty_profile + or existing_env.get("METAFLOW_DEFAULT_METADATA", "") == "service", + abort=False, + ): + env.update(configure_metadata_service(existing_env)) + return env + + +def configure_gs_datastore_and_metadata(existing_env): + empty_profile = False + if not existing_env: + empty_profile = True + env = {} + + # Configure Google Cloud Storage as the datastore. + use_gs_as_datastore = click.confirm( + "\nMetaflow can use " + + yellow("Google Cloud Storage as the storage backend") + + " for all code and data artifacts on " + + "Google Cloud Storage.\nGoogle Cloud Storage is a strict requirement if you " + + "intend to execute your flows on a Kubernetes cluster on GCP (GKE or self-managed)" + + ".\nWould you like to configure Google Cloud Storage " + + "as the default storage backend?", + default=empty_profile + or existing_env.get("METAFLOW_DEFAULT_DATASTORE", "") == "gs", + abort=False, + ) + if use_gs_as_datastore: + env.update(configure_gs_datastore(existing_env)) + + # Configure Metadata service for tracking. + if click.confirm( + "\nMetaflow can use a " + + yellow("remote Metadata Service to track") + + " and persist flow execution metadata.\nConfiguring the " + "service is a requirement if you intend to schedule your " + "flows with Kubernetes on GCP (GKE or self-managed).\nWould you like to " + "configure the Metadata Service?", + default=empty_profile + or existing_env.get("METAFLOW_DEFAULT_METADATA", "") == "service", + abort=False, + ): + env.update(configure_metadata_service(existing_env)) + return env + + +def configure_aws_datastore_and_metadata(existing_env): empty_profile = False if not existing_env: empty_profile = True @@ -664,10 +540,11 @@ def check_kubernetes_client(ctx): import kubernetes except ImportError: echo( - "Please install python kubernetes client first " - + "(run " - + yellow("pip install kubernetes") - + " or equivalent in your favorite python package manager)" + "Could not import module 'Kubernetes'.\nInstall Kubernetes " + + "Python package (https://pypi.org/project/kubernetes/) first.\n" + "You can install the module by executing - \n" + + yellow("%s -m pip install kubernetes" % sys.executable) + + " \nor equivalent in your favorite Python package manager\n" ) ctx.abort() @@ -687,20 +564,20 @@ def check_kubernetes_config(ctx): ) except config.config_exception.ConfigException as e: click.confirm( - "\nYou don't seem to have a valid kubernetes configuration file. " - + "The error from kubernetes client library: " + "\nYou don't seem to have a valid Kubernetes configuration file. " + + "The error from Kubernetes client library: " + red(str(e)) + "." + "To create a kubernetes configuration for EKS, you typically need to run " + yellow("aws eks update-kubeconfig --name ") - + ". For further details, refer to AWS Documentation at https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html\n" - "Do you want to proceed with configuring Metaflow for EKS anyway?", + + ". For further details, refer to AWS documentation at https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html\n" + "Do you want to proceed with configuring Metaflow for Kubernetes anyway?", default=False, abort=True, ) -def configure_eks(existing_env): +def configure_kubernetes(existing_env): empty_profile = False if not existing_env: empty_profile = True @@ -745,6 +622,15 @@ def configure_eks(existing_env): default=existing_env.get("METAFLOW_KUBERNETES_CONTAINER_IMAGE", ""), show_default=True, ) + # Set default Kubernetes secrets to source into pod envs + env["METAFLOW_KUBERNETES_SECRETS"] = click.prompt( + cyan("[METAFLOW_KUBERNETES_SECRETS]") + + yellow(" (optional)") + + " Comma-delimited list of secret names. Jobs will" + " gain environment variables from these secrets. ", + default=existing_env.get("METAFLOW_KUBERNETES_SECRETS", ""), + show_default=True, + ) return env @@ -773,6 +659,144 @@ def verify_aws_credentials(ctx): ctx.abort() +def verify_azure_credentials(ctx): + # Verify that the user has configured AWS credentials on their computer. + if not click.confirm( + "\nMetaflow relies on " + + yellow("Azure access credentials") + + " present on your computer to access resources on Azure." + "\nBefore proceeding further, please confirm that you " + "have already configured these access credentials on " + "this computer.", + default=True, + ): + echo( + "There are many ways to setup your Azure access credentials. You " + "can get started by getting familiar with the following: ", + nl=False, + fg="yellow", + ) + echo("") + echo( + "- https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli", + fg="cyan", + ) + echo( + "- https://docs.microsoft.com/en-us/cli/azure/azure-cli-configuration", + fg="cyan", + ) + ctx.abort() + + +def verify_gcp_credentials(ctx): + # Verify that the user has configured AWS credentials on their computer. + if not click.confirm( + "\nMetaflow relies on " + + yellow("GCP access credentials") + + " present on your computer to access resources on GCP." + "\nBefore proceeding further, please confirm that you " + "have already configured these access credentials on " + "this computer.", + default=True, + ): + echo( + "There are many ways to setup your GCP access credentials. You " + "can get started by getting familiar with the following: ", + nl=False, + fg="yellow", + ) + echo("") + echo( + "- https://cloud.google.com/docs/authentication/provide-credentials-adc", + fg="cyan", + ) + ctx.abort() + + +@configure.command(help="Configure metaflow to access Microsoft Azure.") +@click.option( + "--profile", + "-p", + default="", + help="Configure a named profile. Activate the profile by setting " + "`METAFLOW_PROFILE` environment variable.", +) +@click.pass_context +def azure(ctx, profile): + + # Greet the user! + echo( + "Welcome to Metaflow! Follow the prompts to configure your installation.\n", + bold=True, + ) + + # Check for existing configuration. + if not confirm_overwrite_config(profile): + ctx.abort() + + verify_azure_credentials(ctx) + + existing_env = get_env(profile) + + env = {} + env.update(configure_azure_datastore_and_metadata(existing_env)) + + persist_env({k: v for k, v in env.items() if v}, profile) + + # Prompt user to also configure Kubernetes for compute if using azure + if env.get("METAFLOW_DEFAULT_DATASTORE") == "azure": + click.echo( + "\nFinal note! Metaflow can scale your flows by " + + yellow("executing your steps on Kubernetes.") + + "\nYou may use Azure Kubernetes Service (AKS)" + " or a self-managed Kubernetes cluster on Azure VMs." + + " If/when your Kubernetes cluster is ready for use," + " please run 'metaflow configure kubernetes'.", + ) + + +@configure.command(help="Configure metaflow to access Google Cloud Platform.") +@click.option( + "--profile", + "-p", + default="", + help="Configure a named profile. Activate the profile by setting " + "`METAFLOW_PROFILE` environment variable.", +) +@click.pass_context +def gcp(ctx, profile): + + # Greet the user! + echo( + "Welcome to Metaflow! Follow the prompts to configure your installation.\n", + bold=True, + ) + + # Check for existing configuration. + if not confirm_overwrite_config(profile): + ctx.abort() + + verify_gcp_credentials(ctx) + + existing_env = get_env(profile) + + env = {} + env.update(configure_gs_datastore_and_metadata(existing_env)) + + persist_env({k: v for k, v in env.items() if v}, profile) + + # Prompt user to also configure Kubernetes for compute if using Google Cloud Storage + if env.get("METAFLOW_DEFAULT_DATASTORE") == "gs": + click.echo( + "\nFinal note! Metaflow can scale your flows by " + + yellow("executing your steps on Kubernetes.") + + "\nYou may use Google Kubernetes Engine (GKE)" + " or a self-managed Kubernetes cluster on Google Compute Engine VMs." + + " If/when your Kubernetes cluster is ready for use," + " please run 'metaflow configure kubernetes'.", + ) + + @configure.command(help="Configure metaflow to access self-managed AWS resources.") @click.option( "--profile", @@ -791,7 +815,7 @@ def aws(ctx, profile): ) # Check for existing configuration. - if not overwrite_config(profile): + if not confirm_overwrite_config(profile): ctx.abort() verify_aws_credentials(ctx) @@ -802,7 +826,7 @@ def aws(ctx, profile): empty_profile = True env = {} - env.update(configure_datastore_and_metadata(existing_env)) + env.update(configure_aws_datastore_and_metadata(existing_env)) # Configure AWS Batch for compute if using S3 if env.get("METAFLOW_DEFAULT_DATASTORE") == "s3": @@ -821,7 +845,7 @@ def aws(ctx, profile): persist_env({k: v for k, v in env.items() if v}, profile) -@configure.command(help="Configure metaflow to use AWS EKS.") +@configure.command(help="Configure metaflow to use Kubernetes.") @click.option( "--profile", "-p", @@ -830,7 +854,7 @@ def aws(ctx, profile): "`METAFLOW_PROFILE` environment variable.", ) @click.pass_context -def eks(ctx, profile): +def kubernetes(ctx, profile): check_kubernetes_client(ctx) @@ -843,30 +867,23 @@ def eks(ctx, profile): check_kubernetes_config(ctx) # Check for existing configuration. - if not overwrite_config(profile): + if not confirm_overwrite_config(profile): ctx.abort() - verify_aws_credentials(ctx) - existing_env = get_env(profile) env = existing_env.copy() - if existing_env.get("METAFLOW_DEFAULT_DATASTORE") == "s3": - # Skip S3 configuration if it is already configured - pass - elif not existing_env.get("METAFLOW_DEFAULT_DATASTORE"): - env.update(configure_s3_datastore(existing_env)) - else: - # If configured to use something else, offer to switch to S3 - click.confirm( - "\nMetaflow on EKS needs to use S3 as a datastore, " - + "but your existing configuration is not using S3. " - + "Would you like to reconfigure it to use S3?", - default=True, - abort=True, + # We used to push user straight to S3 configuration inline. + # Now that we support >1 cloud, it gets too complicated. + # Therefore, we instruct the user to configure datastore first, by + # a separate command. + if existing_env.get("METAFLOW_DEFAULT_DATASTORE") == "local": + click.echo( + "\nCannot run Kubernetes with local datastore. Please run" + " 'metaflow configure aws' or 'metaflow configure azure'." ) - env.update(configure_s3_datastore(existing_env)) + click.Abort() # Configure remote metadata. if existing_env.get("METAFLOW_DEFAULT_METADATA") == "service": @@ -883,10 +900,7 @@ def eks(ctx, profile): ): env.update(configure_metadata_service(existing_env)) - # Configure AWS EKS for compute. - env.update(configure_eks(existing_env)) + # Configure Kubernetes for compute. + env.update(configure_kubernetes(existing_env)) persist_env({k: v for k, v in env.items() if v}, profile) - - -main() diff --git a/metaflow/cmd/main_cli.py b/metaflow/cmd/main_cli.py new file mode 100644 index 00000000000..1a976a04c91 --- /dev/null +++ b/metaflow/cmd/main_cli.py @@ -0,0 +1,140 @@ +import os +import traceback + +from metaflow._vendor import click + +from metaflow.plugins.datastores.local_storage import LocalStorage +from metaflow.metaflow_config import DATASTORE_LOCAL_DIR + +from .util import echo_always + + +@click.group() +def main(ctx): + pass + + +@main.command(help="Show all available commands.") +@click.pass_context +def help(ctx): + print(ctx.parent.get_help()) + + +@main.command(help="Show flows accessible from the current working tree.") +def status(): + from metaflow.client import get_metadata + + res = get_metadata() + if res: + res = res.split("@") + else: + raise click.ClickException("Unknown status: cannot find a Metadata provider") + if res[0] == "service": + echo("Using Metadata provider at: ", nl=False) + echo('"%s"\n' % res[1], fg="cyan") + echo("To list available flows, type:\n") + echo("1. python") + echo("2. from metaflow import Metaflow") + echo("3. list(Metaflow())") + return + + from metaflow.client import namespace, metadata, Metaflow + + # Get the local data store path + path = LocalStorage.get_datastore_root_from_config(echo, create_on_absent=False) + # Throw an exception + if path is None: + raise click.ClickException( + "Could not find " + + click.style('"%s"' % DATASTORE_LOCAL_DIR, fg="red") + + " in the current working tree." + ) + + stripped_path = os.path.dirname(path) + namespace(None) + metadata("local@%s" % stripped_path) + echo("Working tree found at: ", nl=False) + echo('"%s"\n' % stripped_path, fg="cyan") + echo("Available flows:", fg="cyan", bold=True) + for flow in Metaflow(): + echo("* %s" % flow, fg="cyan") + + +try: + from metaflow.extension_support import get_modules, load_module, _ext_debug + + _modules_to_import = get_modules("cmd") + _clis = [] + # Reverse to maintain "latest" overrides (in Click, the first one will get it) + for m in reversed(_modules_to_import): + _get_clis = m.module.__dict__.get("get_cmd_clis") + if _get_clis: + _clis.extend(_get_clis()) + +except Exception as e: + _ext_debug("\tWARNING: ignoring all plugins due to error during import: %s" % e) + print( + "WARNING: Command extensions did not load -- ignoring all of them which may not " + "be what you want: %s" % e + ) + _clis = [] + traceback.print_exc() + +from .configure_cmd import cli as configure_cli +from .tutorials_cmd import cli as tutorials_cli + + +@click.command( + cls=click.CommandCollection, + sources=_clis + [main, configure_cli, tutorials_cli], + invoke_without_command=True, +) +@click.pass_context +def start(ctx): + global echo + echo = echo_always + + import metaflow + + echo("Metaflow ", fg="magenta", bold=True, nl=False) + + if ctx.invoked_subcommand is None: + echo("(%s): " % metaflow.__version__, fg="magenta", bold=False, nl=False) + else: + echo("(%s)\n" % metaflow.__version__, fg="magenta", bold=False) + + if ctx.invoked_subcommand is None: + echo("More data science, less engineering\n", fg="magenta") + + # metaflow URL + echo("http://docs.metaflow.org", fg="cyan", nl=False) + echo(" - Read the documentation") + + # metaflow chat + echo("http://chat.metaflow.org", fg="cyan", nl=False) + echo(" - Chat with us") + + # metaflow help email + echo("help@metaflow.org", fg="cyan", nl=False) + echo(" - Get help by email\n") + + print(ctx.get_help()) + + +start() + +for _n in [ + "get_modules", + "load_module", + "_modules_to_import", + "m", + "_get_clis", + "_clis", + "ext_debug", + "e", +]: + try: + del globals()[_n] + except KeyError: + pass +del globals()["_n"] diff --git a/metaflow/cmd/tutorials_cmd.py b/metaflow/cmd/tutorials_cmd.py new file mode 100644 index 00000000000..6e9e68e0789 --- /dev/null +++ b/metaflow/cmd/tutorials_cmd.py @@ -0,0 +1,160 @@ +import os +import shutil + +from metaflow._vendor import click + +from .util import echo_always, makedirs + +echo = echo_always + + +@click.group() +def cli(): + pass + + +@cli.group(help="Browse and access the metaflow tutorial episodes.") +def tutorials(): + pass + + +def get_tutorials_dir(): + metaflow_dir = os.path.dirname(__file__) + package_dir = os.path.dirname(metaflow_dir) + tutorials_dir = os.path.join(package_dir, "metaflow", "tutorials") + + if not os.path.exists(tutorials_dir): + tutorials_dir = os.path.join(package_dir, "tutorials") + + return tutorials_dir + + +def get_tutorial_metadata(tutorial_path): + metadata = {} + with open(os.path.join(tutorial_path, "README.md")) as readme: + content = readme.read() + + paragraphs = [paragraph.strip() for paragraph in content.split("#") if paragraph] + metadata["description"] = paragraphs[0].split("**")[1] + header = paragraphs[0].split("\n") + header = header[0].split(":") + metadata["episode"] = header[0].strip()[len("Episode ") :] + metadata["title"] = header[1].strip() + + for paragraph in paragraphs[1:]: + if paragraph.startswith("Before playing"): + lines = "\n".join(paragraph.split("\n")[1:]) + metadata["prereq"] = lines.replace("```", "") + + if paragraph.startswith("Showcasing"): + lines = "\n".join(paragraph.split("\n")[1:]) + metadata["showcase"] = lines.replace("```", "") + + if paragraph.startswith("To play"): + lines = "\n".join(paragraph.split("\n")[1:]) + metadata["play"] = lines.replace("```", "") + + return metadata + + +def get_all_episodes(): + episodes = [] + for name in sorted(os.listdir(get_tutorials_dir())): + # Skip hidden files (like .gitignore) + if not name.startswith("."): + episodes.append(name) + return episodes + + +@tutorials.command(help="List the available episodes.") +def list(): + echo("Episodes:", fg="cyan", bold=True) + for name in get_all_episodes(): + path = os.path.join(get_tutorials_dir(), name) + metadata = get_tutorial_metadata(path) + echo("* {0: <20} ".format(metadata["episode"]), fg="cyan", nl=False) + echo("- {0}".format(metadata["title"])) + + echo("\nTo pull the episodes, type: ") + echo("metaflow tutorials pull", fg="cyan") + + +def validate_episode(episode): + src_dir = os.path.join(get_tutorials_dir(), episode) + if not os.path.isdir(src_dir): + raise click.BadArgumentUsage( + "Episode " + + click.style('"{0}"'.format(episode), fg="red") + + " does not exist." + " To see a list of available episodes, " + "type:\n" + click.style("metaflow tutorials list", fg="cyan") + ) + + +def autocomplete_episodes(ctx, args, incomplete): + return [k for k in get_all_episodes() if incomplete in k] + + +@tutorials.command(help="Pull episodes " "into your current working directory.") +@click.option( + "--episode", + default="", + help="Optional episode name " "to pull only a single episode.", +) +def pull(episode): + tutorials_dir = get_tutorials_dir() + if not episode: + episodes = get_all_episodes() + else: + episodes = [episode] + # Validate that the list is valid. + for episode in episodes: + validate_episode(episode) + # Create destination `metaflow-tutorials` dir. + dst_parent = os.path.join(os.getcwd(), "metaflow-tutorials") + makedirs(dst_parent) + + # Pull specified episodes. + for episode in episodes: + dst_dir = os.path.join(dst_parent, episode) + # Check if episode has already been pulled before. + if os.path.exists(dst_dir): + if click.confirm( + "Episode " + + click.style('"{0}"'.format(episode), fg="red") + + " has already been pulled before. Do you wish " + "to delete the existing version?" + ): + shutil.rmtree(dst_dir) + else: + continue + echo("Pulling episode ", nl=False) + echo('"{0}"'.format(episode), fg="cyan", nl=False) + # TODO: Is the following redundant? + echo(" into your current working directory.") + # Copy from (local) metaflow package dir to current. + src_dir = os.path.join(tutorials_dir, episode) + shutil.copytree(src_dir, dst_dir) + + echo("\nTo know more about an episode, type:\n", nl=False) + echo("metaflow tutorials info [EPISODE]", fg="cyan") + + +@tutorials.command(help="Find out more about an episode.") +@click.argument("episode", autocompletion=autocomplete_episodes) +def info(episode): + validate_episode(episode) + src_dir = os.path.join(get_tutorials_dir(), episode) + metadata = get_tutorial_metadata(src_dir) + echo("Synopsis:", fg="cyan", bold=True) + echo("%s" % metadata["description"]) + + echo("\nShowcasing:", fg="cyan", bold=True, nl=True) + echo("%s" % metadata["showcase"]) + + if "prereq" in metadata: + echo("\nBefore playing:", fg="cyan", bold=True, nl=True) + echo("%s" % metadata["prereq"]) + + echo("\nTo play:", fg="cyan", bold=True) + echo("%s" % metadata["play"]) diff --git a/metaflow/cmd/util.py b/metaflow/cmd/util.py new file mode 100644 index 00000000000..ceff9869c25 --- /dev/null +++ b/metaflow/cmd/util.py @@ -0,0 +1,23 @@ +import os + +from metaflow._vendor import click + + +def makedirs(path): + # This is for python2 compatibility. + # Python3 has os.makedirs(exist_ok=True). + try: + os.makedirs(path) + except OSError as x: + if x.errno == 17: + return + else: + raise + + +def echo_dev_null(*args, **kwargs): + pass + + +def echo_always(line, **kwargs): + click.secho(line, **kwargs) diff --git a/metaflow/current.py b/metaflow/current.py index 0dc8b15776e..06c58ce05f9 100644 --- a/metaflow/current.py +++ b/metaflow/current.py @@ -14,6 +14,7 @@ def __init__(self): self._origin_run_id = None self._namespace = None self._username = None + self._metadata_str = None self._is_running = False def _raise(ex): @@ -33,7 +34,9 @@ def _set_env( origin_run_id=None, namespace=None, username=None, + metadata_str=None, is_running=True, + tags=None, ): if flow is not None: self._flow_name = flow.name @@ -46,7 +49,9 @@ def _set_env( self._origin_run_id = origin_run_id self._namespace = namespace self._username = username + self._metadata_str = metadata_str self._is_running = is_running + self._tags = tags def _update_env(self, env): for k, v in env.items(): @@ -60,42 +65,151 @@ def get(self, key, default=None): @property def is_running_flow(self): + """ + Returns True if called inside a running Flow, False otherwise. + + You can use this property e.g. inside a library to choose the desired + behavior depending on the execution context. + + Returns + ------- + bool + True if called inside a run, False otherwise. + """ return self._is_running @property def flow_name(self): + """ + The name of the currently executing flow. + + Returns + ------- + str + Flow name. + """ return self._flow_name @property def run_id(self): + """ + The run ID of the currently executing run. + + Returns + ------- + str + Run ID. + """ return self._run_id @property def step_name(self): + """ + The name of the currently executing step. + + Returns + ------- + str + Step name. + """ return self._step_name @property def task_id(self): + """ + The task ID of the currently executing task. + + Returns + ------- + str + Task ID. + """ return self._task_id @property def retry_count(self): + """ + The index of the task execution attempt. + + This property returns 0 for the first attempt to execute the task. + If the @retry decorator is used and the first attempt fails, this + property returns the number of times the task was attempted prior + to the current attempt. + + Returns + ------- + int + The retry count. + """ return self._retry_count @property def origin_run_id(self): + """ + The run ID of the original run this run was resumed from. + + This property returns None for ordinary runs. If the run + was started by the resume command, the property returns + the ID of the original run. + + You can use this property to detect if the run is resumed + or not. + + Returns + ------- + str + Run ID of the original run. + """ return self._origin_run_id @property def pathspec(self): - return "/".join((self._flow_name, self._run_id, self._step_name, self._task_id)) + """ + Pathspec of the current run, i.e. a unique + identifier of the current task. The returned + string follows this format: + ``` + {flow_name}/{run_id}/{step_name}/{task_id} + ``` + + Returns + ------- + str + Pathspec. + """ + + pathspec_components = ( + self._flow_name, + self._run_id, + self._step_name, + self._task_id, + ) + if any(v is None for v in pathspec_components): + return None + return "/".join(pathspec_components) @property def namespace(self): + """ + The current namespace. + + Returns + ------- + str + Namespace. + """ return self._namespace @property def username(self): + """ + The name of the user who started the run, if available. + + Returns + ------- + str + User name. + """ return self._username @property @@ -106,6 +220,15 @@ def parallel(self): node_index=int(os.environ.get("MF_PARALLEL_NODE_INDEX", "0")), ) + @property + def tags(self): + """ + [Legacy function - do not use] + + Access tags through the Run object instead. + """ + return self._tags + # instantiate the Current singleton. This will be populated # by task.MetaflowTask before a task is executed. diff --git a/metaflow/datastore/__init__.py b/metaflow/datastore/__init__.py index 794ea23ccef..793251b0cff 100644 --- a/metaflow/datastore/__init__.py +++ b/metaflow/datastore/__init__.py @@ -2,8 +2,3 @@ from .flow_datastore import FlowDataStore from .datastore_set import TaskDataStoreSet from .task_datastore import TaskDataStore - -from .local_storage import LocalStorage -from .s3_storage import S3Storage - -DATASTORES = {"local": LocalStorage, "s3": S3Storage} diff --git a/metaflow/datastore/datastore_storage.py b/metaflow/datastore/datastore_storage.py index dfbbd0ef2c0..e8fad3c2bb1 100644 --- a/metaflow/datastore/datastore_storage.py +++ b/metaflow/datastore/datastore_storage.py @@ -229,6 +229,8 @@ def save_bytes(self, path_and_bytes_iter, overwrite=False, len_hint=0): BufferedIOBase. overwrite : bool True if the objects can be overwritten. Defaults to False. + Even when False, it is NOT an error condition to see an existing object. + Simply do not perform the upload operation. len_hint : int Estimated number of items produced by the iterator @@ -260,7 +262,7 @@ def load_bytes(self, keys): Metadata will be None if no metadata is present; otherwise it is a dictionary of metadata associated with the object. - Note that the file at `file_path` may no longer be accessible outside of + Note that the file at `file_path` may no longer be accessible outside the scope of the returned object. The order of items in the list is not to be relied on (ie: rely on the key diff --git a/metaflow/datastore/flow_datastore.py b/metaflow/datastore/flow_datastore.py index 4dfc6321f8f..f2707c10166 100644 --- a/metaflow/datastore/flow_datastore.py +++ b/metaflow/datastore/flow_datastore.py @@ -89,7 +89,7 @@ def get_latest_task_datastores( must also be specified, by default None pathspecs : List[str], optional Full task specs (run_id/step_name/task_id). Can be used instead of - specifiying run_id and steps, by default None + specifying run_id and steps, by default None allow_not_done : bool, optional If True, returns the latest attempt of a task even if that attempt wasn't marked as done, by default False @@ -204,7 +204,7 @@ def save_data(self, data_iter, len_hint=0): Parameters ---------- - data : Iterator[bytes] + data_iter : Iterator[bytes] Iterator over blobs to save; each item in the list will be saved individually. len_hint : int Estimate of the number of items that will be produced by the iterator, diff --git a/metaflow/datastore/task_datastore.py b/metaflow/datastore/task_datastore.py index f99856682f9..3f003173a0f 100644 --- a/metaflow/datastore/task_datastore.py +++ b/metaflow/datastore/task_datastore.py @@ -16,6 +16,8 @@ from .exceptions import DataException, UnpicklableArtifactException +_included_file_type = "" + def only_if_not_done(f): @wraps(f) @@ -150,7 +152,7 @@ def __init__( if self._attempt is None: self._attempt = max_attempt elif max_attempt is None or self._attempt > max_attempt: - # In this case, the attempt does not exist so we can't load + # In this case the attempt does not exist, so we can't load # anything self._objects = {} self._info = {} @@ -299,6 +301,7 @@ def pickle_iter(): "type": str(type(obj)), "encoding": encode_type, } + artifact_names.append(name) yield blob @@ -386,7 +389,11 @@ def get_artifact_sizes(self, names): """ for name in names: info = self._info.get(name) - yield name, info.get("size", 0) + if info["type"] == _included_file_type: + sz = self[name].size + else: + sz = info.get("size", 0) + yield name, sz @require_mode("r") def get_legacy_log_size(self, stream): @@ -569,7 +576,7 @@ def is_none(self, name): # Conservatively check if the actual object is None, # in case the artifact is stored using a different python version. # Note that if an object is None and stored in Py2 and accessed in - # Py3, this test will fail and we will fallback to the slow path. This + # Py3, this test will fail and we will fall back to the slow path. This # is intended (being conservative) if obj_type == str(type(None)): return True @@ -681,7 +688,7 @@ def persist(self, flow): self._info.update(flow._datastore._info) # we create a list of valid_artifacts in advance, outside of - # artifacts_iter so we can provide a len_hint below + # artifacts_iter, so we can provide a len_hint below valid_artifacts = [] for var in dir(flow): if var.startswith("__") or var in flow._EPHEMERAL: @@ -783,18 +790,37 @@ def to_dict(self, show_private=False, max_value_size=None, include=None): continue if k[0] == "_" and not show_private: continue - if max_value_size is not None and self._info[k]["size"] > max_value_size: - d[k] = ArtifactTooLarge() + + info = self._info[k] + if max_value_size is not None: + if info["type"] == _included_file_type: + sz = self[k].size + else: + sz = info.get("size", 0) + + if sz == 0 or sz > max_value_size: + d[k] = ArtifactTooLarge() + else: + d[k] = self[k] + if info["type"] == _included_file_type: + d[k] = d[k].decode(k) else: d[k] = self[k] + if info["type"] == _included_file_type: + d[k] = d[k].decode(k) + return d @require_mode("r") def format(self, **kwargs): def lines(): for k, v in self.to_dict(**kwargs).items(): + if self._info[k]["type"] == _included_file_type: + sz = self[k].size + else: + sz = self._info[k]["size"] yield k, "*{key}* [size: {size} type: {type}] = {value}".format( - key=k, value=v, **self._info[k] + key=k, value=v, size=sz, type=self._info[k]["type"] ) return "\n".join(line for k, line in sorted(lines())) diff --git a/metaflow/datatools/s3.py b/metaflow/datatools/s3.py deleted file mode 100644 index a55f41b9e02..00000000000 --- a/metaflow/datatools/s3.py +++ /dev/null @@ -1,1047 +0,0 @@ -import json -import os -import sys -import time -import shutil -import random -import subprocess -from io import RawIOBase, BytesIO, BufferedIOBase -from itertools import chain, starmap -from tempfile import mkdtemp, NamedTemporaryFile - -from .. import FlowSpec -from ..current import current -from ..metaflow_config import DATATOOLS_S3ROOT, S3_RETRY_COUNT -from ..util import ( - namedtuple_with_defaults, - is_stringish, - to_bytes, - to_unicode, - to_fileobj, - url_quote, - url_unquote, -) -from ..exception import MetaflowException -from ..debug import debug - -try: - # python2 - from urlparse import urlparse -except: - # python3 - from urllib.parse import urlparse - -from .s3util import get_s3_client, read_in_chunks, get_timestamp - -try: - import boto3 - from boto3.s3.transfer import TransferConfig - - DOWNLOAD_FILE_THRESHOLD = 2 * TransferConfig().multipart_threshold - DOWNLOAD_MAX_CHUNK = 2 * 1024 * 1024 * 1024 - 1 - boto_found = True -except: - boto_found = False - - -def ensure_unicode(x): - return None if x is None else to_unicode(x) - - -S3GetObject = namedtuple_with_defaults("S3GetObject", "key offset length") - -S3PutObject = namedtuple_with_defaults( - "S3PutObject", - "key value path content_type metadata", - defaults=(None, None, None, None), -) - -RangeInfo = namedtuple_with_defaults( - "RangeInfo", "total_size request_offset request_length", defaults=(0, -1) -) - - -class MetaflowS3InvalidObject(MetaflowException): - headline = "Not a string-like object" - - -class MetaflowS3URLException(MetaflowException): - headline = "Invalid address" - - -class MetaflowS3Exception(MetaflowException): - headline = "S3 access failed" - - -class MetaflowS3NotFound(MetaflowException): - headline = "S3 object not found" - - -class MetaflowS3AccessDenied(MetaflowException): - headline = "S3 access denied" - - -class S3Object(object): - """ - This object represents a path or an object in S3, - with an optional local copy. - Get or list calls return one or more of S3Objects. - """ - - def __init__( - self, - prefix, - url, - path, - size=None, - content_type=None, - metadata=None, - range_info=None, - last_modified=None, - ): - - # all fields of S3Object should return a unicode object - prefix, url, path = map(ensure_unicode, (prefix, url, path)) - - self._size = size - self._url = url - self._path = path - self._key = None - self._content_type = content_type - self._last_modified = last_modified - - self._metadata = None - if metadata is not None and "metaflow-user-attributes" in metadata: - self._metadata = json.loads(metadata["metaflow-user-attributes"]) - - if range_info and ( - range_info.request_length is None or range_info.request_length < 0 - ): - self._range_info = RangeInfo( - range_info.total_size, range_info.request_offset, range_info.total_size - ) - else: - self._range_info = range_info - - if path: - self._size = os.stat(self._path).st_size - - if prefix is None or prefix == url: - self._key = url - self._prefix = None - else: - self._key = url[len(prefix.rstrip("/")) + 1 :].rstrip("/") - self._prefix = prefix - - @property - def exists(self): - """ - Does this key correspond to an object in S3? - """ - return self._size is not None - - @property - def downloaded(self): - """ - Has this object been downloaded? - """ - return bool(self._path) - - @property - def url(self): - """ - S3 location of the object - """ - return self._url - - @property - def prefix(self): - """ - Prefix requested that matches the object. - """ - return self._prefix - - @property - def key(self): - """ - Key corresponds to the key given to the get call that produced - this object. This may be a full S3 URL or a suffix based on what - was requested. - """ - return self._key - - @property - def path(self): - """ - Path to the local file corresponding to the object downloaded. - This file gets deleted automatically when a S3 scope exits. - Returns None if this S3Object has not been downloaded. - """ - return self._path - - @property - def blob(self): - """ - Contents of the object as a byte string. - Returns None if this S3Object has not been downloaded. - """ - if self._path: - with open(self._path, "rb") as f: - return f.read() - - @property - def text(self): - """ - Contents of the object as a Unicode string. - Returns None if this S3Object has not been downloaded. - """ - if self._path: - return self.blob.decode("utf-8", errors="replace") - - @property - def size(self): - """ - Size of the object in bytes. - Returns None if the key does not correspond to an object in S3. - """ - return self._size - - @property - def has_info(self): - """ - Returns true if this S3Object contains the content-type or user-metadata. - If False, this means that content_type and range_info will not return the - proper information - """ - return self._content_type is not None or self._metadata is not None - - @property - def metadata(self): - """ - Returns a dictionary of user-defined metadata - """ - return self._metadata - - @property - def content_type(self): - """ - Returns the content-type of the S3 object; if unknown, returns None - """ - return self._content_type - - @property - def range_info(self): - """ - Returns a namedtuple containing the following fields: - - total_size: size in S3 of the object - - request_offset: the starting offset in this S3Object - - request_length: the length in this S3Object - """ - return self._range_info - - @property - def last_modified(self): - """ - Returns the last modified unix timestamp of the object, or None - if not fetched. - """ - return self._last_modified - - def __str__(self): - if self._path: - return "" % (self._url, self._size) - elif self._size: - return "" % (self._url, self._size) - else: - return "" % self._url - - def __repr__(self): - return str(self) - - -class S3Client(object): - def __init__(self): - self._s3_client = None - self._s3_error = None - - @property - def client(self): - if self._s3_client is None: - self.reset_client() - return self._s3_client - - @property - def error(self): - if self._s3_error is None: - self.reset_client() - return self._s3_error - - def reset_client(self): - self._s3_client, self._s3_error = get_s3_client() - - -class S3(object): - @classmethod - def get_root_from_config(cls, echo, create_on_absent=True): - return DATATOOLS_S3ROOT - - def __init__( - self, tmproot=".", bucket=None, prefix=None, run=None, s3root=None, **kwargs - ): - """ - Initialize a new context for S3 operations. This object is used as - a context manager for a with statement. - There are two ways to initialize this object depending whether you want - to bind paths to a Metaflow run or not. - 1. With a run object: - run: (required) Either a FlowSpec object (typically 'self') or a - Run object corresponding to an existing Metaflow run. These - are used to add a version suffix in the S3 path. - bucket: (optional) S3 bucket. - prefix: (optional) S3 prefix. - 2. Without a run object: - s3root: (optional) An S3 root URL for all operations. If this is - not specified, all operations require a full S3 URL. - These options are supported in both the modes: - tmproot: (optional) Root path for temporary files (default: '.') - """ - - if not boto_found: - raise MetaflowException("You need to install 'boto3' in order to use S3.") - - if run: - # 1. use a (current) run ID with optional customizations - parsed = urlparse(DATATOOLS_S3ROOT) - if not bucket: - bucket = parsed.netloc - if not prefix: - prefix = parsed.path - if isinstance(run, FlowSpec): - if current.is_running_flow: - prefix = os.path.join(prefix, current.flow_name, current.run_id) - else: - raise MetaflowS3URLException( - "Initializing S3 with a FlowSpec outside of a running " - "flow is not supported." - ) - else: - prefix = os.path.join(prefix, run.parent.id, run.id) - - self._s3root = u"s3://%s" % os.path.join(bucket, prefix.strip("/")) - elif s3root: - # 2. use an explicit S3 prefix - parsed = urlparse(to_unicode(s3root)) - if parsed.scheme != "s3": - raise MetaflowS3URLException( - "s3root needs to be an S3 URL prefxied with s3://." - ) - self._s3root = s3root.rstrip("/") - else: - # 3. use the client only with full URLs - self._s3root = None - - self._s3_client = kwargs.get("external_client", S3Client()) - self._tmpdir = mkdtemp(dir=tmproot, prefix="metaflow.s3.") - - def __enter__(self): - return self - - def __exit__(self, *args): - self.close() - - def close(self): - """ - Delete all temporary files downloaded in this context. - """ - try: - if not debug.s3client: - if self._tmpdir: - shutil.rmtree(self._tmpdir) - self._tmpdir = None - except: - pass - - def _url(self, key_value): - # NOTE: All URLs are handled as Unicode objects (unicode in py2, - # string in py3) internally. We expect that all URLs passed to this - # class as either Unicode or UTF-8 encoded byte strings. All URLs - # returned are Unicode. - key = getattr(key_value, "key", key_value) - if self._s3root is None: - parsed = urlparse(to_unicode(key)) - if parsed.scheme == "s3" and parsed.path: - return key - else: - if current.is_running_flow: - raise MetaflowS3URLException( - "Specify S3(run=self) when you use S3 inside a running " - "flow. Otherwise you have to use S3 with full " - "s3:// urls." - ) - else: - raise MetaflowS3URLException( - "Initialize S3 with an 's3root' or 'run' if you don't " - "want to specify full s3:// urls." - ) - elif key: - if key.startswith("s3://"): - raise MetaflowS3URLException( - "Don't use absolute S3 URLs when the S3 client is " - "initialized with a prefix. URL: %s" % key - ) - return os.path.join(self._s3root, key) - else: - return self._s3root - - def _url_and_range(self, key_value): - url = self._url(key_value) - start = getattr(key_value, "offset", None) - length = getattr(key_value, "length", None) - range_str = None - # Range specification are inclusive so getting from offset 500 for 100 - # bytes will read as bytes=500-599 - if start is not None or length is not None: - if start is None: - start = 0 - if length is None: - # Fetch from offset till the end of the file - range_str = "bytes=%d-" % start - elif length < 0: - # Fetch from end; ignore start value here - range_str = "bytes=-%d" % (-length) - else: - # Typical range fetch - range_str = "bytes=%d-%d" % (start, start + length - 1) - return url, range_str - - def list_paths(self, keys=None): - """ - List the next level of paths in S3. If multiple keys are - specified, listings are done in parallel. The returned - S3Objects have .exists == False if the url refers to a - prefix, not an existing S3 object. - Args: - keys: (required) a list of suffixes for paths to list. - Returns: - a list of S3Objects (not downloaded) - Example: - Consider the following paths in S3: - A/B/C - D/E - In this case, list_paths(['A', 'D']), returns ['A/B', 'D/E']. The - first S3Object has .exists == False, since it does not refer to an - object in S3. It is just a prefix. - """ - - def _list(keys): - if keys is None: - keys = [None] - urls = ((self._url(key).rstrip("/") + "/", None) for key in keys) - res = self._read_many_files("list", urls) - for s3prefix, s3url, size in res: - if size: - yield s3prefix, s3url, None, int(size) - else: - yield s3prefix, s3url, None, None - - return list(starmap(S3Object, _list(keys))) - - def list_recursive(self, keys=None): - """ - List objects in S3 recursively. If multiple keys are - specified, listings are done in parallel. The returned - S3Objects have always .exists == True, since they refer - to existing objects in S3. - Args: - keys: (required) a list of suffixes for paths to list. - Returns: - a list of S3Objects (not downloaded) - Example: - Consider the following paths in S3: - A/B/C - D/E - In this case, list_recursive(['A', 'D']), returns ['A/B/C', 'D/E']. - """ - - def _list(keys): - if keys is None: - keys = [None] - res = self._read_many_files( - "list", map(self._url_and_range, keys), recursive=True - ) - for s3prefix, s3url, size in res: - yield s3prefix, s3url, None, int(size) - - return list(starmap(S3Object, _list(keys))) - - def info(self, key=None, return_missing=False): - """ - Get information about a single object from S3 - Args: - key: (optional) a suffix identifying the object. - return_missing: (optional, default False) if set to True, do - not raise an exception for a missing key but - return it as an S3Object with .exists == False. - Returns: - an S3Object containing information about the object. The - downloaded property will be false and exists will indicate whether - or not the file exists - """ - url = self._url(key) - src = urlparse(url) - - def _info(s3, tmp): - resp = s3.head_object(Bucket=src.netloc, Key=src.path.lstrip('/"')) - return { - "content_type": resp["ContentType"], - "metadata": resp["Metadata"], - "size": resp["ContentLength"], - "last_modified": get_timestamp(resp["LastModified"]), - } - - info_results = None - try: - _, info_results = self._one_boto_op(_info, url, create_tmp_file=False) - except MetaflowS3NotFound: - if return_missing: - info_results = None - else: - raise - if info_results: - return S3Object( - self._s3root, - url, - path=None, - size=info_results["size"], - content_type=info_results["content_type"], - metadata=info_results["metadata"], - last_modified=info_results["last_modified"], - ) - return S3Object(self._s3root, url, None) - - def info_many(self, keys, return_missing=False): - """ - Get information about many objects from S3 in parallel. - Args: - keys: (required) a list of suffixes identifying the objects. - return_missing: (optional, default False) if set to True, do - not raise an exception for a missing key but - return it as an S3Object with .exists == False. - Returns: - a list of S3Objects corresponding to the objects requested. The - downloaded property will be false and exists will indicate whether - or not the file exists. - """ - - def _head(): - from . import s3op - - res = self._read_many_files( - "info", map(self._url_and_range, keys), verbose=False, listing=True - ) - - for s3prefix, s3url, fname in res: - if fname: - # We have a metadata file to read from - with open(os.path.join(self._tmpdir, fname), "r") as f: - info = json.load(f) - if info["error"] is not None: - # We have an error, we check if it is a missing file - if info["error"] == s3op.ERROR_URL_NOT_FOUND: - if return_missing: - yield self._s3root, s3url, None - else: - raise MetaflowS3NotFound() - elif info["error"] == s3op.ERROR_URL_ACCESS_DENIED: - raise MetaflowS3AccessDenied() - else: - raise MetaflowS3Exception("Got error: %d" % info["error"]) - else: - yield self._s3root, s3url, None, info["size"], info[ - "content_type" - ], info["metadata"], None, info["last_modified"] - else: - # This should not happen; we should always get a response - # even if it contains an error inside it - raise MetaflowS3Exception("Did not get a response to HEAD") - - return list(starmap(S3Object, _head())) - - def get(self, key=None, return_missing=False, return_info=True): - """ - Get a single object from S3. - Args: - key: (optional) a suffix identifying the object. Can also be - an object containing the properties `key`, `offset` and - `length` to specify a range query. `S3GetObject` is such an object. - return_missing: (optional, default False) if set to True, do - not raise an exception for a missing key but - return it as an S3Object with .exists == False. - return_info: (optional, default True) if set to True, fetch the - content-type and user metadata associated with the object. - Returns: - an S3Object corresponding to the object requested. - """ - url, r = self._url_and_range(key) - src = urlparse(url) - - def _download(s3, tmp): - if r: - resp = s3.get_object( - Bucket=src.netloc, Key=src.path.lstrip("/"), Range=r - ) - else: - resp = s3.get_object(Bucket=src.netloc, Key=src.path.lstrip("/")) - sz = resp["ContentLength"] - if not r and sz > DOWNLOAD_FILE_THRESHOLD: - # In this case, it is more efficient to use download_file as it - # will download multiple parts in parallel (it does it after - # multipart_threshold) - s3.download_file(src.netloc, src.path.lstrip("/"), tmp) - else: - with open(tmp, mode="wb") as t: - read_in_chunks(t, resp["Body"], sz, DOWNLOAD_MAX_CHUNK) - if return_info: - return { - "content_type": resp["ContentType"], - "metadata": resp["Metadata"], - "last_modified": get_timestamp(resp["LastModified"]), - } - return None - - addl_info = None - try: - path, addl_info = self._one_boto_op(_download, url) - except MetaflowS3NotFound: - if return_missing: - path = None - else: - raise - if addl_info: - return S3Object( - self._s3root, - url, - path, - content_type=addl_info["content_type"], - metadata=addl_info["metadata"], - last_modified=addl_info["last_modified"], - ) - return S3Object(self._s3root, url, path) - - def get_many(self, keys, return_missing=False, return_info=True): - """ - Get many objects from S3 in parallel. - Args: - keys: (required) a list of suffixes identifying the objects. Each - item in the list can also be an object containing the properties - `key`, `offset` and `length to specify a range query. - `S3GetObject` is such an object. - return_missing: (optional, default False) if set to True, do - not raise an exception for a missing key but - return it as an S3Object with .exists == False. - return_info: (optional, default True) if set to True, fetch the - content-type and user metadata associated with the object. - Returns: - a list of S3Objects corresponding to the objects requested. - """ - - def _get(): - res = self._read_many_files( - "get", - map(self._url_and_range, keys), - allow_missing=return_missing, - verify=True, - verbose=False, - info=return_info, - listing=True, - ) - - for s3prefix, s3url, fname in res: - if return_info: - if fname: - # We have a metadata file to read from - with open( - os.path.join(self._tmpdir, "%s_meta" % fname), "r" - ) as f: - info = json.load(f) - yield self._s3root, s3url, os.path.join( - self._tmpdir, fname - ), None, info["content_type"], info["metadata"], None, info[ - "last_modified" - ] - else: - yield self._s3root, s3prefix, None - else: - if fname: - yield self._s3root, s3url, os.path.join(self._tmpdir, fname) - else: - # missing entries per return_missing=True - yield self._s3root, s3prefix, None - - return list(starmap(S3Object, _get())) - - def get_recursive(self, keys, return_info=False): - """ - Get many objects from S3 recursively in parallel. - Args: - keys: (required) a list of suffixes for paths to download - recursively. - return_info: (optional, default False) if set to True, fetch the - content-type and user metadata associated with the object. - Returns: - a list of S3Objects corresponding to the objects requested. - """ - - def _get(): - res = self._read_many_files( - "get", - map(self._url_and_range, keys), - recursive=True, - verify=True, - verbose=False, - info=return_info, - listing=True, - ) - - for s3prefix, s3url, fname in res: - if return_info: - # We have a metadata file to read from - with open(os.path.join(self._tmpdir, "%s_meta" % fname), "r") as f: - info = json.load(f) - yield self._s3root, s3url, os.path.join( - self._tmpdir, fname - ), None, info["content_type"], info["metadata"], None, info[ - "last_modified" - ] - else: - yield s3prefix, s3url, os.path.join(self._tmpdir, fname) - - return list(starmap(S3Object, _get())) - - def get_all(self, return_info=False): - """ - Get all objects from S3 recursively (in parallel). This request - only works if S3 is initialized with a run or a s3root prefix. - Args: - return_info: (optional, default False) if set to True, fetch the - content-type and user metadata associated with the object. - Returns: - a list of S3Objects corresponding to the objects requested. - """ - if self._s3root is None: - raise MetaflowS3URLException( - "Can't get_all() when S3 is initialized without a prefix" - ) - else: - return self.get_recursive([None], return_info) - - def put(self, key, obj, overwrite=True, content_type=None, metadata=None): - """ - Put an object to S3. - Args: - key: (required) suffix for the object. - obj: (required) a bytes, string, or a unicode object to - be stored in S3. - overwrite: (optional) overwrites the key with obj, if it exists - content_type: (optional) string representing the MIME type of the - object - metadata: (optional) User metadata to store alongside the object - Returns: - an S3 URL corresponding to the object stored. - """ - if isinstance(obj, (RawIOBase, BufferedIOBase)): - if not obj.readable() or not obj.seekable(): - raise MetaflowS3InvalidObject( - "Object corresponding to the key '%s' is not readable or seekable" - % key - ) - blob = obj - else: - if not is_stringish(obj): - raise MetaflowS3InvalidObject( - "Object corresponding to the key '%s' is not a string " - "or a bytes object." % key - ) - blob = to_fileobj(obj) - # We override the close functionality to prevent closing of the - # file if it is used multiple times when uploading (since upload_fileobj - # will/may close it on failure) - real_close = blob.close - blob.close = lambda: None - - url = self._url(key) - src = urlparse(url) - extra_args = None - if content_type or metadata: - extra_args = {} - if content_type: - extra_args["ContentType"] = content_type - if metadata: - extra_args["Metadata"] = { - "metaflow-user-attributes": json.dumps(metadata) - } - - def _upload(s3, _): - # We make sure we are at the beginning in case we are retrying - blob.seek(0) - s3.upload_fileobj( - blob, src.netloc, src.path.lstrip("/"), ExtraArgs=extra_args - ) - - if overwrite: - self._one_boto_op(_upload, url, create_tmp_file=False) - real_close() - return url - else: - - def _head(s3, _): - s3.head_object(Bucket=src.netloc, Key=src.path.lstrip("/")) - - try: - self._one_boto_op(_head, url, create_tmp_file=False) - except MetaflowS3NotFound: - self._one_boto_op(_upload, url, create_tmp_file=False) - finally: - real_close() - return url - - def put_many(self, key_objs, overwrite=True): - """ - Put objects to S3 in parallel. - Args: - key_objs: (required) an iterator of (key, value) tuples. Value must - be a string, bytes, or a unicode object. Instead of - (key, value) tuples, you can also pass any object that - has the following properties 'key', 'value', 'content_type', - 'metadata' like the S3PutObject for example. 'key' and - 'value' are required but others are optional. - overwrite: (optional) overwrites the key with obj, if it exists - Returns: - a list of (key, S3 URL) tuples corresponding to the files sent. - """ - - def _store(): - for key_obj in key_objs: - if isinstance(key_obj, tuple): - key = key_obj[0] - obj = key_obj[1] - else: - key = key_obj.key - obj = key_obj.value - store_info = { - "key": key, - "content_type": getattr(key_obj, "content_type", None), - } - metadata = getattr(key_obj, "metadata", None) - if metadata: - store_info["metadata"] = { - "metaflow-user-attributes": json.dumps(metadata) - } - if isinstance(obj, (RawIOBase, BufferedIOBase)): - if not obj.readable() or not obj.seekable(): - raise MetaflowS3InvalidObject( - "Object corresponding to the key '%s' is not readable or seekable" - % key - ) - else: - if not is_stringish(obj): - raise MetaflowS3InvalidObject( - "Object corresponding to the key '%s' is not a string " - "or a bytes object." % key - ) - obj = to_fileobj(obj) - with NamedTemporaryFile( - dir=self._tmpdir, - delete=False, - mode="wb", - prefix="metaflow.s3.put_many.", - ) as tmp: - tmp.write(obj.read()) - tmp.close() - yield tmp.name, self._url(key), store_info - - return self._put_many_files(_store(), overwrite) - - def put_files(self, key_paths, overwrite=True): - """ - Put files to S3 in parallel. - Args: - key_paths: (required) an iterator of (key, path) tuples. Instead of - (key, path) tuples, you can also pass any object that - has the following properties 'key', 'path', 'content_type', - 'metadata' like the S3PutObject for example. 'key' and - 'path' are required but others are optional. - overwrite: (optional) overwrites the key with obj, if it exists - Returns: - a list of (key, S3 URL) tuples corresponding to the files sent. - """ - - def _check(): - for key_path in key_paths: - if isinstance(key_path, tuple): - key = key_path[0] - path = key_path[1] - else: - key = key_path.key - path = key_path.path - store_info = { - "key": key, - "content_type": getattr(key_path, "content_type", None), - } - metadata = getattr(key_path, "metadata", None) - if metadata: - store_info["metadata"] = { - "metaflow-user-attributes": json.dumps(metadata) - } - if not os.path.exists(path): - raise MetaflowS3NotFound("Local file not found: %s" % path) - yield path, self._url(key), store_info - - return self._put_many_files(_check(), overwrite) - - def _one_boto_op(self, op, url, create_tmp_file=True): - error = "" - for i in range(S3_RETRY_COUNT + 1): - tmp = None - if create_tmp_file: - tmp = NamedTemporaryFile( - dir=self._tmpdir, prefix="metaflow.s3.one_file.", delete=False - ) - try: - side_results = op(self._s3_client.client, tmp.name if tmp else None) - return tmp.name if tmp else None, side_results - except self._s3_client.error as err: - from . import s3op - - error_code = s3op.normalize_client_error(err) - if error_code == 404: - raise MetaflowS3NotFound(url) - elif error_code == 403: - raise MetaflowS3AccessDenied(url) - elif error_code == "NoSuchBucket": - raise MetaflowS3URLException("Specified S3 bucket doesn't exist.") - error = str(err) - except Exception as ex: - # TODO specific error message for out of disk space - error = str(ex) - if tmp: - os.unlink(tmp.name) - self._s3_client.reset_client() - # add some jitter to make sure retries are not synchronized - time.sleep(2 ** i + random.randint(0, 10)) - raise MetaflowS3Exception( - "S3 operation failed.\n" "Key requested: %s\n" "Error: %s" % (url, error) - ) - - # NOTE: re: _read_many_files and _put_many_files - # All file IO is through binary files - we write bytes, we read - # bytes. All inputs and outputs from these functions are Unicode. - # Conversion between bytes and unicode is done through - # and url_unquote. - def _read_many_files(self, op, prefixes_and_ranges, **options): - prefixes_and_ranges = list(prefixes_and_ranges) - with NamedTemporaryFile( - dir=self._tmpdir, - mode="wb", - delete=not debug.s3client, - prefix="metaflow.s3.inputs.", - ) as inputfile: - inputfile.write( - b"\n".join( - [ - b" ".join([url_quote(prefix)] + ([url_quote(r)] if r else [])) - for prefix, r in prefixes_and_ranges - ] - ) - ) - inputfile.flush() - stdout, stderr = self._s3op_with_retries( - op, inputs=inputfile.name, **options - ) - if stderr: - raise MetaflowS3Exception( - "Getting S3 files failed.\n" - "First prefix requested: %s\n" - "Error: %s" % (prefixes_and_ranges[0], stderr) - ) - else: - for line in stdout.splitlines(): - yield tuple(map(url_unquote, line.strip(b"\n").split(b" "))) - - def _put_many_files(self, url_info, overwrite): - url_info = list(url_info) - url_dicts = [ - dict( - chain([("local", os.path.realpath(local)), ("url", url)], info.items()) - ) - for local, url, info in url_info - ] - - with NamedTemporaryFile( - dir=self._tmpdir, - mode="wb", - delete=not debug.s3client, - prefix="metaflow.s3.put_inputs.", - ) as inputfile: - lines = [to_bytes(json.dumps(x)) for x in url_dicts] - inputfile.write(b"\n".join(lines)) - inputfile.flush() - stdout, stderr = self._s3op_with_retries( - "put", - filelist=inputfile.name, - verbose=False, - overwrite=overwrite, - listing=True, - ) - if stderr: - raise MetaflowS3Exception( - "Uploading S3 files failed.\n" - "First key: %s\n" - "Error: %s" % (url_info[0][2]["key"], stderr) - ) - else: - urls = set() - for line in stdout.splitlines(): - url, _, _ = map(url_unquote, line.strip(b"\n").split(b" ")) - urls.add(url) - return [(info["key"], url) for _, url, info in url_info if url in urls] - - def _s3op_with_retries(self, mode, **options): - from . import s3op - - cmdline = [sys.executable, os.path.abspath(s3op.__file__), mode] - for key, value in options.items(): - key = key.replace("_", "-") - if isinstance(value, bool): - if value: - cmdline.append("--%s" % key) - else: - cmdline.append("--no-%s" % key) - else: - cmdline.extend(("--%s" % key, value)) - - for i in range(S3_RETRY_COUNT + 1): - with NamedTemporaryFile( - dir=self._tmpdir, - mode="wb+", - delete=not debug.s3client, - prefix="metaflow.s3op.stderr", - ) as stderr: - try: - debug.s3client_exec(cmdline) - stdout = subprocess.check_output( - cmdline, cwd=self._tmpdir, stderr=stderr.file - ) - return stdout, None - except subprocess.CalledProcessError as ex: - stderr.seek(0) - err_out = stderr.read().decode("utf-8", errors="replace") - stderr.seek(0) - if ex.returncode == s3op.ERROR_URL_NOT_FOUND: - raise MetaflowS3NotFound(err_out) - elif ex.returncode == s3op.ERROR_URL_ACCESS_DENIED: - raise MetaflowS3AccessDenied(err_out) - print("Error with S3 operation:", err_out) - time.sleep(2 ** i + random.randint(0, 10)) - - return None, err_out diff --git a/metaflow/debug.py b/metaflow/debug.py index c4b1e4ac03d..fbfaca95dc8 100644 --- a/metaflow/debug.py +++ b/metaflow/debug.py @@ -1,4 +1,5 @@ from __future__ import print_function +import inspect import sys from functools import partial @@ -22,7 +23,7 @@ def __init__(self): import metaflow.metaflow_config as config for typ in config.DEBUG_OPTIONS: - if getattr(config, "METAFLOW_DEBUG_%s" % typ.upper()): + if getattr(config, "DEBUG_%s" % typ.upper()): op = partial(self.log, typ) else: op = self.noop @@ -37,7 +38,9 @@ def log(self, typ, args): s = args else: s = " ".join(args) - print("debug[%s]: %s" % (typ, s), file=sys.stderr) + lineno = inspect.currentframe().f_back.f_lineno + filename = inspect.stack()[1][1] + print("debug[%s %s:%s]: %s" % (typ, filename, lineno, s), file=sys.stderr) def noop(self, args): pass diff --git a/metaflow/decorators.py b/metaflow/decorators.py index 0042e71dcaa..8723367b79b 100644 --- a/metaflow/decorators.py +++ b/metaflow/decorators.py @@ -1,4 +1,5 @@ from functools import partial +import json import re import os import sys @@ -13,6 +14,13 @@ from metaflow._vendor import click +try: + unicode +except NameError: + unicode = str + basestring = str + + class BadStepDecoratorException(MetaflowException): headline = "Syntax error" @@ -114,21 +122,42 @@ def __init__(self, attributes=None, statically_defined=False): @classmethod def _parse_decorator_spec(cls, deco_spec): - top = deco_spec.split(":", 1) - if len(top) == 1: + if len(deco_spec) == 0: return cls() - else: - name, attrspec = top - attrs = dict( - map(lambda x: x.strip(), a.split("=")) - for a in re.split(""",(?=[\s\w]+=)""", attrspec.strip("\"'")) - ) - return cls(attributes=attrs) + + attrs = {} + # TODO: Do we really want to allow spaces in the names of attributes?!? + for a in re.split(""",(?=[\s\w]+=)""", deco_spec): + name, val = a.split("=", 1) + try: + val_parsed = json.loads(val.strip().replace('\\"', '"')) + except json.JSONDecodeError: + # In this case, we try to convert to either an int or a float or + # leave as is. Prefer ints if possible. + try: + val_parsed = int(val.strip()) + except ValueError: + try: + val_parsed = float(val.strip()) + except ValueError: + val_parsed = val.strip() + + attrs[name.strip()] = val_parsed + return cls(attributes=attrs) def make_decorator_spec(self): attrs = {k: v for k, v in self.attributes.items() if v is not None} if attrs: - attrstr = ",".join("%s=%s" % x for x in attrs.items()) + attr_list = [] + # We dump simple types directly as string to get around the nightmare quote + # escaping but for more complex types (typically dictionaries or lists), + # we dump using JSON. + for k, v in attrs.items(): + if isinstance(v, (int, float, unicode, basestring)): + attr_list.append("%s=%s" % (k, str(v))) + else: + attr_list.append("%s=%s" % (k, json.dumps(v).replace('"', '\\"'))) + attrstr = ",".join(attr_list) return "%s:%s" % (self.name, attrstr) else: return self.name @@ -148,7 +177,7 @@ class FlowDecorator(Decorator): options = {} def __init__(self, *args, **kwargs): - # Note that this assumes we are executing one flow per process so we have a global list of + # Note that this assumes we are executing one flow per process, so we have a global list of # _flow_decorators. A similar setup is used in parameters. self._flow_decorators.append(self) super(FlowDecorator, self).__init__(*args, **kwargs) @@ -167,7 +196,7 @@ def get_top_level_options(self): options that should be passed to subprocesses (tasks). The option names should be a subset of the keys in self.options. - If the decorator has a non-empty set of options in self.options, you + If the decorator has a non-empty set of options in `self.options`, you probably want to return the assigned values in this method. """ return [] @@ -450,8 +479,10 @@ def _attach_decorators_to_step(step, decospecs): from .plugins import STEP_DECORATORS decos = {decotype.name: decotype for decotype in STEP_DECORATORS} + for decospec in decospecs: - deconame = decospec.strip("'").split(":")[0] + splits = decospec.split(":", 1) + deconame = splits[0] if deconame not in decos: raise UnknownStepDecoratorException(deconame) # Attach the decorator to step if it doesn't have the decorator @@ -461,8 +492,11 @@ def _attach_decorators_to_step(step, decospecs): deconame not in [deco.name for deco in step.decorators] or decos[deconame].allow_multiple ): - # if the decorator is present in a step and is of type allow_mutliple then add the decorator to the step - deco = decos[deconame]._parse_decorator_spec(decospec) + # if the decorator is present in a step and is of type allow_multiple + # then add the decorator to the step + deco = decos[deconame]._parse_decorator_spec( + splits[1] if len(splits) > 1 else "" + ) step.decorators.append(deco) diff --git a/metaflow/event_logger.py b/metaflow/event_logger.py index 7e559c443fd..aee08d6902a 100644 --- a/metaflow/event_logger.py +++ b/metaflow/event_logger.py @@ -1,33 +1,29 @@ -from .sidecar import SidecarSubProcess -from .sidecar_messages import Message, MessageTypes +from metaflow.sidecar import Message, MessageTypes, Sidecar class NullEventLogger(object): + TYPE = "nullSidecarLogger" + def __init__(self, *args, **kwargs): - pass + # Currently passed flow and env in kwargs + self._sidecar = Sidecar(self.TYPE) def start(self): - pass - - def log(self, payload): - pass + return self._sidecar.start() def terminate(self): - pass - + return self._sidecar.terminate() -class EventLogger(NullEventLogger): - def __init__(self, logger_type): - # type: (str) -> None - self.sidecar_process = None - self.logger_type = logger_type - - def start(self): - self.sidecar_process = SidecarSubProcess(self.logger_type) + def send(self, msg): + # Arbitrary message sending. Useful if you want to override some different + # types of messages. + self._sidecar.send(msg) def log(self, payload): - msg = Message(MessageTypes.LOG_EVENT, payload) - self.sidecar_process.msg_handler(msg) + if self._sidecar.is_active: + msg = Message(MessageTypes.BEST_EFFORT, payload) + self._sidecar.send(msg) - def terminate(self): - self.sidecar_process.kill() + @classmethod + def get_worker(cls): + return None diff --git a/metaflow/exception.py b/metaflow/exception.py index f3552bc3340..fb2a48689ca 100644 --- a/metaflow/exception.py +++ b/metaflow/exception.py @@ -95,6 +95,10 @@ class MetaflowInternalError(MetaflowException): headline = "Internal error" +class MetaflowTaggingError(MetaflowException): + headline = "Tagging error" + + class MetaflowUnknownUser(MetaflowException): headline = "Unknown user" diff --git a/metaflow/extension_support.py b/metaflow/extension_support.py index b6d04935570..59203c1a112 100644 --- a/metaflow/extension_support.py +++ b/metaflow/extension_support.py @@ -1,3 +1,5 @@ +from __future__ import print_function + import importlib import json import os @@ -7,24 +9,76 @@ from collections import defaultdict, namedtuple +from importlib.abc import MetaPathFinder, Loader from itertools import chain +# +# This file provides the support for Metaflow's extension mechanism which allows +# a Metaflow developer to extend metaflow by providing a package `metaflow_extensions`. +# Multiple such packages can be provided, and they will all be loaded into Metaflow in a +# way that is transparent to the user. +# +# NOTE: The conventions used here may change over time and this is an advanced feature. +# +# The general functionality provided here can be divided into three phases: +# - Package discovery: in this part, packages that provide metaflow extensions +# are discovered. This is contained in the `_get_extension_packages` function +# - Integration with Metaflow: throughout the Metaflow code, extension points +# are provided (they are given below in `_extension_points`). At those points, +# the core Metaflow code will invoke functions to load the packages discovered +# in the first phase. These functions are: +# - get_modules: Returns all modules that are contributing to the extension +# point; this is typically done first. +# - load_module: Simple loading of a specific module +# - load_globals: Utility function to load the globals from a module into +# another globals()-like object +# - alias_submodules: Determines the aliases for modules allowing metaflow.Z to alias +# metaflow_extensions.X.Y.Z for example. This supports the __mf_promote_submodules__ +# construct as well as aliasing any modules present in the extension. This is +# typically used in conjunction with lazy_load_aliases which takes care of actually +# making the aliasing work lazily (ie: modules that are not already loaded are only +# loaded on use). +# - lazy_load_aliases: Adds loaders for all the module aliases produced by +# alias_submodules for example +# - multiload_globals: Convenience function to `load_globals` on all modules returned +# by `get_modules` +# - multiload_all: Convenience function to `load_globals` and +# `lazy_load_aliases(alias_submodules()) on all modules returned by `get_modules` +# - Packaging the extensions: when extensions need to be included in the code package, +# this allows the extensions to be properly included (including potentially non .py +# files). To support this: +# - dump_module_info dumps information in the INFO file allowing packaging to work +# in a Conda environment or a remote environment (it saves file paths, load order, etc) +# - package_mfext_package: allows the packaging of a single extension +# - package_mfext_all: packages all extensions +# +# The get_aliases_modules is used by Pylint to ignore some of the errors arising from +# aliasing packages + __all__ = ( "load_module", "get_modules", "dump_module_info", + "get_aliased_modules", + "package_mfext_package", + "package_mfext_all", "load_globals", "alias_submodules", "EXT_PKG", "lazy_load_aliases", "multiload_globals", "multiload_all", + "_ext_debug", ) EXT_PKG = "metaflow_extensions" EXT_CONFIG_REGEXP = re.compile(r"^mfextinit_[a-zA-Z0-9_-]+\.py$") +EXT_META_REGEXP = re.compile(r"^mfextmeta_[a-zA-Z0-9_-]+\.py$") +REQ_NAME = re.compile(r"^(([a-zA-Z0-9][a-zA-Z0-9._-]*[a-zA-Z0-9])|[a-zA-Z0-9]).*$") +EXT_EXCLUDE_SUFFIXES = [".pyc"] -METAFLOW_DEBUG_EXT_MECHANISM = os.environ.get("METAFLOW_DEBUG_EXT", False) +# To get verbose messages, set METAFLOW_DEBUG_EXT to 1 +DEBUG_EXT = os.environ.get("METAFLOW_DEBUG_EXT", False) MFExtPackage = namedtuple("MFExtPackage", "package_name tl_package config_module") @@ -47,33 +101,72 @@ def get_modules(extension_point): ) _ext_debug("Getting modules for extension point '%s'..." % extension_point) for pkg in _pkgs_per_extension_point.get(extension_point, []): - _ext_debug("\tFound TL '%s' from '%s'" % (pkg.tl_package, pkg.package_name)) - m = _get_extension_config(pkg.tl_package, extension_point, pkg.config_module) + _ext_debug(" Found TL '%s' from '%s'" % (pkg.tl_package, pkg.package_name)) + m = _get_extension_config( + pkg.package_name, pkg.tl_package, extension_point, pkg.config_module + ) if m: modules_to_load.append(m) - _ext_debug("\tLoaded %s" % str(modules_to_load)) + _ext_debug(" Loaded %s" % str(modules_to_load)) return modules_to_load def dump_module_info(): - return "ext_info", [_all_packages, _pkgs_per_extension_point] + _filter_files_all() + sanitized_all_packages = dict() + # Strip out root_paths (we don't need it and no need to expose user's dir structure) + for k, v in _all_packages.items(): + sanitized_all_packages[k] = { + "root_paths": None, + "meta_module": v["meta_module"], + "files": v["files"], + } + return "ext_info", [sanitized_all_packages, _pkgs_per_extension_point] + + +def get_aliased_modules(): + return _aliased_modules + + +def package_mfext_package(package_name): + from metaflow.util import to_unicode + + _ext_debug("Packaging '%s'" % package_name) + _filter_files_package(package_name) + pkg_info = _all_packages.get(package_name, None) + if pkg_info and pkg_info.get("root_paths", None): + single_path = len(pkg_info["root_paths"]) == 1 + for p in pkg_info["root_paths"]: + root_path = to_unicode(p) + for f in pkg_info["files"]: + f_unicode = to_unicode(f) + fp = os.path.join(root_path, f_unicode) + if single_path or os.path.isfile(fp): + _ext_debug(" Adding '%s'" % fp) + yield fp, os.path.join(EXT_PKG, f_unicode) + + +def package_mfext_all(): + for p in _all_packages: + for path_tuple in package_mfext_package(p): + yield path_tuple def load_globals(module, dst_globals, extra_indent=False): if extra_indent: - extra_indent = "\t" + extra_indent = " " else: extra_indent = "" _ext_debug("%sLoading globals from '%s'" % (extra_indent, module.__name__)) for n, o in module.__dict__.items(): if not n.startswith("__") and not isinstance(o, types.ModuleType): - _ext_debug("%s\tImporting '%s'" % (extra_indent, n)) + _ext_debug("%s Importing '%s'" % (extra_indent, n)) dst_globals[n] = o def alias_submodules(module, tl_package, extension_point, extra_indent=False): if extra_indent: - extra_indent = "\t" + extra_indent = " " else: extra_indent = "" lazy_load_custom_modules = {} @@ -107,7 +200,7 @@ def alias_submodules(module, tl_package, extension_point, extra_indent=False): ) if lazy_load_custom_modules: _ext_debug( - "%s\tFound explicit promotions in __mf_promote_submodules__: %s" + "%s Found explicit promotions in __mf_promote_submodules__: %s" % (extra_indent, str(list(lazy_load_custom_modules.keys()))) ) for n, o in module.__dict__.items(): @@ -123,15 +216,16 @@ def alias_submodules(module, tl_package, extension_point, extra_indent=False): else: lazy_load_custom_modules["metaflow.%s" % n] = o _ext_debug( - "%s\tWill create the following module aliases: %s" + "%s Will create the following module aliases: %s" % (extra_indent, str(list(lazy_load_custom_modules.keys()))) ) + _aliased_modules.extend(lazy_load_custom_modules.keys()) return lazy_load_custom_modules def lazy_load_aliases(aliases): if aliases: - sys.meta_path = [_LazyLoader(aliases)] + sys.meta_path + sys.meta_path = [_LazyFinder(aliases)] + sys.meta_path def multiload_globals(modules, dst_globals): @@ -141,7 +235,7 @@ def multiload_globals(modules, dst_globals): def multiload_all(modules, extension_point, dst_globals): for m in modules: - # Note that we load aliases separately (as opposed to ine one fell swoop) so + # Note that we load aliases separately (as opposed to in one fell swoop) so # modules loaded later in `modules` can depend on them lazy_load_aliases( alias_submodules(m.module, m.tl_package, extension_point, extra_indent=True) @@ -149,44 +243,50 @@ def multiload_all(modules, extension_point, dst_globals): load_globals(m.module, dst_globals) -_py_ver = sys.version_info[0] * 10 + sys.version_info[1] +_py_ver = sys.version_info[:2] _mfext_supported = False +_aliased_modules = [] -if _py_ver >= 34: +if _py_ver >= (3, 4): import importlib.util - from importlib.machinery import ModuleSpec - if _py_ver >= 38: + if _py_ver >= (3, 8): from importlib import metadata + elif _py_ver >= (3, 6): + from metaflow._vendor.v3_6 import importlib_metadata as metadata else: - from metaflow._vendor import importlib_metadata as metadata + from metaflow._vendor.v3_5 import importlib_metadata as metadata _mfext_supported = True -else: - # Something random so there is no syntax error - ModuleSpec = None -# IMPORTANT: More specific paths must appear FIRST (before any less specific one) +# Extension points are the directories that can be present in a EXT_PKG to +# contribute to that extension point. For example, if you have +# metaflow_extensions/X/plugins, your extension contributes to the plugins +# extension point. +# IMPORTANT: More specific paths must appear FIRST (before any less specific one). For +# efficiency, put the less specific ones directly under more specific ones. _extension_points = [ "plugins.env_escape", "plugins.cards", + "plugins.datatools", + "plugins", "config", - "datatools", "exceptions", - "plugins", "toplevel", + "cmd", ] def _ext_debug(*args, **kwargs): - if METAFLOW_DEBUG_EXT_MECHANISM: + if DEBUG_EXT: init_str = "%s:" % EXT_PKG + kwargs["file"] = sys.stderr print(init_str, *args, **kwargs) def _get_extension_packages(): if not _mfext_supported: _ext_debug("Not supported for your Python version -- 3.4+ is needed") - return [], {} + return {}, {} # If we have an INFO file with the appropriate information (if running from a saved # code package for example), we use that directly @@ -194,7 +294,7 @@ def _get_extension_packages(): from metaflow import INFO_FILE try: - with open(INFO_FILE, "r") as contents: + with open(INFO_FILE, encoding="utf-8") as contents: all_pkg, ext_to_pkg = json.load(contents).get("ext_info", (None, None)) if all_pkg is not None and ext_to_pkg is not None: _ext_debug("Loading pre-computed information from INFO file") @@ -210,13 +310,22 @@ def _get_extension_packages(): try: extensions_module = importlib.import_module(EXT_PKG) except ImportError as e: - if _py_ver >= 36: + if _py_ver >= (3, 6): # e.name is set to the name of the package that fails to load # so don't error ONLY IF the error is importing this module (but do # error if there is a transitive import error) if not (isinstance(e, ModuleNotFoundError) and e.name == EXT_PKG): raise - return [], {} + return {}, {} + + # There are two "types" of packages: + # - those installed on the system (distributions) + # - those present in the PYTHONPATH + # We have more information on distributions (including dependencies) and more + # effective ways to get file information from them (they include the full list of + # files installed) so we treat them separately from packages purely in PYTHONPATH. + # They are also the more likely way that users will have extensions present, so + # we optimize for that case. # At this point, we look at all the paths and create a set. As we find distributions # that match it, we will remove from the set and then will be left with any @@ -227,54 +336,117 @@ def _get_extension_packages(): list_ext_points = [x.split(".") for x in _extension_points] init_ext_points = [x[0] for x in list_ext_points] - # TODO: This relies only on requirements to determine import order; we may want + # NOTE: For distribution packages, we will rely on requirements to determine the + # load order of extensions: if distribution A and B both provide EXT_PKG and + # distribution A depends on B then when returning modules in `get_modules`, we will + # first return B and THEN A. We may want # other ways of specifying "load me after this if it exists" without depending on # the package. One way would be to rely on the description and have that info there. - # Not sure of the use though so maybe we can skip for now. - mf_ext_packages = [] - # Key: distribution name/full path to package - # Value: - # Key: TL package name - # Value: MFExtPackage + # Not sure of the use, though, so maybe we can skip for now. + + # Key: distribution name/package path + # Value: Dict containing: + # root_paths: The root path for all the files in this package. Can be a list in + # some rare cases + # meta_module: The module to the meta file (if any) that contains information about + # how to package this extension (suffixes to include/exclude) + # files: The list of files to be included (or considered for inclusion) when + # packaging this extension + mf_ext_packages = dict() + + # Key: extension point (one of _extension_point) + # Value: another dictionary with + # Key: distribution name/full path to package + # Value: another dictionary with + # Key: TL package name (so in metaflow_extensions.X...., the X) + # Value: MFExtPackage extension_points_to_pkg = defaultdict(dict) + + # Key: string: configuration file for a package + # Value: list: packages that this configuration file is present in config_to_pkg = defaultdict(list) + # Same as config_to_pkg for meta files + meta_to_pkg = defaultdict(list) + + # 1st step: look for distributions (the common case) for dist in metadata.distributions(): if any( [pkg == EXT_PKG for pkg in (dist.read_text("top_level.txt") or "").split()] ): + if dist.metadata["Name"] in mf_ext_packages: + _ext_debug( + "Ignoring duplicate package '%s' (duplicate paths in sys.path? (%s))" + % (dist.metadata["Name"], str(sys.path)) + ) + continue _ext_debug("Found extension package '%s'..." % dist.metadata["Name"]) # Remove the path from the paths to search. This is not 100% accurate because - # it is possible that at that same location there is a package and a non - # package but it is exceedingly unlikely so we are going to ignore this. - all_paths.discard(dist.locate_file(EXT_PKG).as_posix()) + # it is possible that at that same location there is a package and a non-package, + # but it is exceedingly unlikely, so we are going to ignore this. + dist_root = dist.locate_file(EXT_PKG).as_posix() + all_paths.discard(dist_root) - mf_ext_packages.append(dist.metadata["Name"]) + files_to_include = [] + meta_module = None # At this point, we check to see what extension points this package # contributes to. This is to enable multiple namespace packages to contribute # to the same extension point (for example, you may have multiple packages # that have plugins) for f in dist.files: - # Make sure EXT_PKG is a ns package - if f.as_posix() == "%s/__init__.py" % EXT_PKG: - raise RuntimeError( - "Package '%s' providing '%s' is not an implicit namespace " - "package as required" % (dist.metadata["Name"], EXT_PKG) - ) - parts = list(f.parts) - if ( - len(parts) > 1 - and parts[0] == EXT_PKG - and parts[1] in init_ext_points - ): - # This is most likely a problem as we need an intermediate "identifier" - raise RuntimeError( - "Package '%s' should conform to %s.X.%s and not %s.%s where " - "X is your organization's name for example" - % (dist.metadata["Name"], EXT_PKG, parts[1], EXT_PKG, parts[1]) - ) + + if len(parts) > 1 and parts[0] == EXT_PKG: + # Ensure that we don't have a __init__.py to force this package to + # be a NS package + if parts[1] == "__init__.py": + raise RuntimeError( + "Package '%s' providing '%s' is not an implicit namespace " + "package as required" % (dist.metadata["Name"], EXT_PKG) + ) + + # Record the file as a candidate for inclusion when packaging if + # needed + if not any( + parts[-1].endswith(suffix) for suffix in EXT_EXCLUDE_SUFFIXES + ): + files_to_include.append(os.path.join(*parts[1:])) + + if parts[1] in init_ext_points: + # This is most likely a problem as we need an intermediate + # "identifier" + raise RuntimeError( + "Package '%s' should conform to '%s.X.%s' and not '%s.%s' where " + "X is your organization's name for example" + % ( + dist.metadata["Name"], + EXT_PKG, + parts[1], + EXT_PKG, + parts[1], + ) + ) + + # Check for any metadata; we can only have one metadata per + # distribution at most + if EXT_META_REGEXP.match(parts[1]) is not None: + potential_meta_module = ".".join([EXT_PKG, parts[1][:-3]]) + if meta_module: + raise RuntimeError( + "Package '%s' defines more than one meta configuration: " + "'%s' and '%s' (at least)" + % ( + dist.metadata["Name"], + meta_module, + potential_meta_module, + ) + ) + meta_module = potential_meta_module + _ext_debug( + "Found meta '%s' for '%s'" % (meta_module, dist_full_name) + ) + meta_to_pkg[meta_module].append(dist_full_name) if len(parts) > 3 and parts[0] == EXT_PKG: # We go over _extension_points *in order* to make sure we get more @@ -290,9 +462,9 @@ def _get_extension_packages(): # Check if this is an "init" file config_module = None - if ( - len(parts) == len(ext_list) + 3 - and EXT_CONFIG_REGEXP.match(parts[-1]) is not None + if len(parts) == len(ext_list) + 3 and ( + EXT_CONFIG_REGEXP.match(parts[-1]) is not None + or parts[-1] == "__init__.py" ): parts[-1] = parts[-1][:-3] # Remove the .py config_module = ".".join(parts) @@ -320,42 +492,48 @@ def _get_extension_packages(): ) if config_module is not None: _ext_debug( - "\tTL %s found config file '%s'" + " TL '%s' found config file '%s'" % (parts[1], config_module) ) extension_points_to_pkg[_extension_points[idx]][ dist.metadata["Name"] ][parts[1]] = MFExtPackage( - package_name=dist_full_name, + package_name=dist.metadata["Name"], tl_package=parts[1], config_module=config_module, ) else: _ext_debug( - "\tTL %s extends '%s' with config '%s'" + " TL '%s' extends '%s' with config '%s'" % (parts[1], _extension_points[idx], config_module) ) extension_points_to_pkg[_extension_points[idx]][ dist.metadata["Name"] ][parts[1]] = MFExtPackage( - package_name=dist_full_name, + package_name=dist.metadata["Name"], tl_package=parts[1], config_module=config_module, ) break - + mf_ext_packages[dist.metadata["Name"]] = { + "root_paths": [dist_root], + "meta_module": meta_module, + "files": files_to_include, + } # At this point, we have all the packages that contribute to EXT_PKG, # we now check to see if there is an order to respect based on dependencies. We will # return an ordered list that respects that order and is ordered alphabetically in # case of ties. We do not do any checks because we rely on pip to have done those. + # Basically topological sort based on dependencies. pkg_to_reqs_count = {} req_to_dep = {} - mf_ext_packages_set = set(mf_ext_packages) for pkg_name in mf_ext_packages: req_count = 0 - req_pkgs = [x.split()[0] for x in metadata.requires(pkg_name) or []] + req_pkgs = [ + REQ_NAME.match(x).group(1) for x in metadata.requires(pkg_name) or [] + ] for req_pkg in req_pkgs: - if req_pkg in mf_ext_packages_set: + if req_pkg in mf_ext_packages: req_count += 1 req_to_dep.setdefault(req_pkg, []).append(pkg_name) pkg_to_reqs_count[pkg_name] = req_count @@ -389,112 +567,217 @@ def _get_extension_packages(): # Check that we got them all if len(pkg_to_reqs_count) > 0: raise RuntimeError( - "Unresolved dependencies in %s: %s" % (EXT_PKG, str(pkg_to_reqs_count)) + "Unresolved dependencies in '%s': %s" + % (EXT_PKG, ", and ".join("'%s'" % p for p in pkg_to_reqs_count)) ) + _ext_debug("'%s' distributions order is %s" % (EXT_PKG, str(mf_pkg_list))) + # We check if we have any additional packages that were not yet installed that - # we need to use. We always put them *last*. - if len(all_paths) > 0: + # we need to use. We always put them *last* in the load order and put them + # alphabetically. + all_paths_list = list(all_paths) + all_paths_list.sort() + + # This block of code is the equivalent of the one above for distributions except + # for PYTHONPATH packages. The functionality is identical, but it looks a little + # different because we construct the file list instead of having it nicely provided + # to us. + package_name_to_path = dict() + if len(all_paths_list) > 0: _ext_debug("Non installed packages present at %s" % str(all_paths)) - packages_to_add = set() - for package_path in all_paths: - _ext_debug("Walking path %s" % package_path) + for package_count, package_path in enumerate(all_paths_list): + # We give an alternate name for the visible package name. It is + # not exposed to the end user but used to refer to the package, and it + # doesn't provide much additional information to have the full path + # particularly when it is on a remote machine. + # We keep a temporary mapping around for error messages while loading for + # the first time. + package_name = "_pythonpath_%d" % package_count + _ext_debug( + "Walking path %s (package name %s)" % (package_path, package_name) + ) + package_name_to_path[package_name] = package_path base_depth = len(package_path.split("/")) + files_to_include = [] + meta_module = None for root, dirs, files in os.walk(package_path): parts = root.split("/") cur_depth = len(parts) + # relative_root strips out metaflow_extensions + relative_root = "/".join(parts[base_depth:]) + relative_module = ".".join(parts[base_depth - 1 :]) + files_to_include.extend( + [ + "/".join([relative_root, f]) if relative_root else f + for f in files + if not any( + [f.endswith(suffix) for suffix in EXT_EXCLUDE_SUFFIXES] + ) + ] + ) if cur_depth == base_depth: if "__init__.py" in files: raise RuntimeError( - "%s at '%s' is not an implicit namespace package as required" + "'%s' at '%s' is not an implicit namespace package as required" % (EXT_PKG, root) ) for d in dirs: if d in init_ext_points: raise RuntimeError( - "Package at %s should conform to %s.X.%s and not %s.%s " - "where X is your organization's name for example" + "Package at '%s' should conform to' %s.X.%s' and not " + "'%s.%s' where X is your organization's name for example" % (root, EXT_PKG, d, EXT_PKG, d) ) + # Check for meta files for this package + meta_files = [ + x for x in map(EXT_META_REGEXP.match, files) if x is not None + ] + if meta_files: + # We should have one meta file at most + if len(meta_files) > 1: + raise RuntimeError( + "Package at '%s' defines more than one meta file: %s" + % ( + package_path, + ", and ".join( + ["'%s'" % x.group(0) for x in meta_files] + ), + ) + ) + else: + meta_module = ".".join( + [relative_module, meta_files[0].group(0)[:-3]] + ) + elif cur_depth > base_depth + 1: # We want at least a TL name and something under tl_name = parts[base_depth] - tl_fullname = "/".join([package_path, tl_name]) + tl_fullname = "%s[%s]" % (package_path, tl_name) prefix_match = parts[base_depth + 1 :] - next_dirs = None for idx, ext_list in enumerate(list_ext_points): if prefix_match == ext_list: + # We check to see if this is an actual extension point + # or if we just have a directory on the way to another + # extension point. To do this, we check to see if we have + # any files or directories that are *not* directly another + # extension point + skip_extension = len(files) == 0 + if skip_extension: + next_dir_idx = len(list_ext_points[idx]) + ok_subdirs = [ + list_ext_points[j][next_dir_idx] + for j in range(0, idx) + if len(list_ext_points[j]) > next_dir_idx + ] + skip_extension = set(dirs).issubset(set(ok_subdirs)) + + if skip_extension: + _ext_debug( + " Skipping '%s' as no files/directory of interest" + % _extension_points[idx] + ) + continue + # Check for any "init" files init_files = [ - x + x.group(0) for x in map(EXT_CONFIG_REGEXP.match, files) if x is not None ] + if "__init__.py" in files: + init_files.append("__init__.py") + config_module = None if len(init_files) > 1: raise RuntimeError( - "Package at %s defines more than one configuration " + "Package at '%s' defines more than one configuration " "file for '%s': %s" % ( tl_fullname, ".".join(prefix_match), - ", and ".join( - ["'%s'" % x.group(0) for x in init_files] - ), + ", and ".join(["'%s'" % x for x in init_files]), ) ) elif len(init_files) == 1: config_module = ".".join( - parts[base_depth - 1 :] - + [init_files[0].group(0)[:-3]] + [relative_module, init_files[0][:-3]] ) config_to_pkg[config_module].append(tl_fullname) + d = extension_points_to_pkg[_extension_points[idx]][ - tl_fullname + package_name ] = dict() d[tl_name] = MFExtPackage( - package_name=tl_fullname, + package_name=package_name, tl_package=tl_name, config_module=config_module, ) _ext_debug( - "\tExtends '%s' with config '%s'" + " Extends '%s' with config '%s'" % (_extension_points[idx], config_module) ) - packages_to_add.add(tl_fullname) - else: - # Check what directories we need to go down if any - if len(ext_list) > 1 and prefix_match == ext_list[:-1]: - if next_dirs is None: - next_dirs = [] - next_dirs.append(ext_list[-1]) - if next_dirs is not None: - dirs[:] = next_dirs[:] - - # Add all these new packages to the list of packages as well. - packages_to_add = list(packages_to_add) - packages_to_add.sort() - mf_pkg_list.extend(packages_to_add) - - # Sanity check that we only have one package per configuration file + mf_pkg_list.append(package_name) + mf_ext_packages[package_name] = { + "root_paths": [package_path], + "meta_module": meta_module, + "files": files_to_include, + } + + # Sanity check that we only have one package per configuration file. + # This prevents multiple packages from providing the same named configuration + # file which would result in one overwriting the other if they are both installed. errors = [] for m, packages in config_to_pkg.items(): if len(packages) > 1: errors.append( - "\tPackages %s define the same configuration module '%s'" - % (", and ".join(packages), m) + " Packages %s define the same configuration module '%s'" + % (", and ".join(["'%s'" % p for p in packages]), m) + ) + for m, packages in meta_to_pkg.items(): + if len(packages) > 1: + errors.append( + " Packages %s define the same meta module '%s'" + % (", and ".join(["'%s'" % p for p in packages]), m) ) if errors: raise RuntimeError( - "Conflicts in %s configuration files:\n%s" % (EXT_PKG, "\n".join(errors)) + "Conflicts in '%s' files:\n%s" % (EXT_PKG, "\n".join(errors)) ) extension_points_to_pkg.default_factory = None - # Figure out the per extension point order + + # We have the load order globally; we now figure it out per extension point. for k, v in extension_points_to_pkg.items(): + + # v is a dict distributionName/packagePath -> (dict tl_name -> MFPackage) l = [v[pkg].values() for pkg in mf_pkg_list if pkg in v] - extension_points_to_pkg[k] = list(chain(*l)) - return mf_pkg_list, extension_points_to_pkg + # In the case of the plugins.cards extension we allow those packages + # to be ns packages, so we only list the package once (in its first position). + # In all other cases, we error out if we don't have a configuration file for the + # package (either a __init__.py of an explicit mfextinit_*.py) + final_list = [] + null_config_tl_package = set() + for pkg in chain(*l): + if pkg.config_module is None: + if k == "plugins.cards": + # This is allowed here but we only keep one + if pkg.tl_package in null_config_tl_package: + continue + null_config_tl_package.add(pkg.tl_package) + else: + package_path = package_name_to_path.get(pkg.package_name) + if package_path: + package_path = "at '%s'" % package_path + else: + package_path = "'%s'" % pkg.package_name + raise RuntimeError( + "Package %s does not define a configuration file for '%s'" + % (package_path, k) + ) + final_list.append(pkg) + extension_points_to_pkg[k] = final_list + return mf_ext_packages, extension_points_to_pkg _all_packages, _pkgs_per_extension_point = _get_extension_packages() @@ -504,7 +787,7 @@ def _attempt_load_module(module_name): try: extension_module = importlib.import_module(module_name) except ImportError as e: - if _py_ver >= 36: + if _py_ver >= (3, 6): # e.name is set to the name of the package that fails to load # so don't error ONLY IF the error is importing this module (but do # error if there is a transitive import error) @@ -514,122 +797,303 @@ def _attempt_load_module(module_name): errored_names.append("%s.%s" % (errored_names[-1], p)) if not (isinstance(e, ModuleNotFoundError) and e.name in errored_names): print( - "The following exception occured while trying to load %s ('%s')" + "The following exception occurred while trying to load '%s' ('%s')" % (EXT_PKG, module_name) ) raise else: - _ext_debug("\t\tUnknown error when loading '%s': %s" % (module_name, e)) + _ext_debug( + " Unknown error when loading '%s': %s" % (module_name, e) + ) return None else: return extension_module -def _get_extension_config(tl_pkg, extension_point, config_module): - module_name = ".".join([EXT_PKG, tl_pkg, extension_point]) - if config_module is not None: - _ext_debug("\t\tAttempting to load '%s'" % config_module) - extension_module = _attempt_load_module(config_module) +def _get_extension_config(distribution_name, tl_pkg, extension_point, config_module): + if config_module is not None and not config_module.endswith("__init__"): + module_name = config_module + # file_path below will be /root/metaflow_extensions/X/Y/mfextinit_Z.py and + # module name is metaflow_extensions.X.Y.mfextinit_Z so if we want to strip to + # /root/metaflow_extensions, we need to remove this number of elements from the + # filepath + strip_from_filepath = len(module_name.split(".")) - 1 else: - _ext_debug("\t\tAttempting to load '%s'" % module_name) - extension_module = _attempt_load_module(module_name) + module_name = ".".join([EXT_PKG, tl_pkg, extension_point]) + # file_path here will be /root/metaflow_extensions/X/Y/__init__.py BUT + # module name is metaflow_extensions.X.Y so we have a 1 off compared to the + # previous case + strip_from_filepath = len(module_name.split(".")) + + _ext_debug(" Attempting to load '%s'" % module_name) + + extension_module = _attempt_load_module(module_name) + if extension_module: + # We update the path to this module. This is useful if we need to package this + # package again. Note that in most cases, packaging happens in the outermost + # local python environment (non Conda and not remote) so we already have the + # root_paths set when we are initially looking for metaflow_extensions package. + # This code allows for packaging while running inside a Conda environment or + # remotely where the root_paths has been changed since the initial packaging. + # This currently does not happen much. + if _all_packages[distribution_name]["root_paths"] is None: + file_path = getattr(extension_module, "__file__") + if file_path: + # Common case where this is an actual init file (mfextinit_X.py or __init__.py) + root_paths = ["/".join(file_path.split("/")[:-strip_from_filepath])] + else: + # Only used for plugins.cards where the package can be a NS package. In + # this case, __path__ will have things like /root/metaflow_extensions/X/Y + # and module name will be metaflow_extensions.X.Y + root_paths = [ + "/".join(p.split("/")[: -len(module_name.split(".")) + 1]) + for p in extension_module.__path__ + ] + + _ext_debug("Package '%s' is rooted at %s" % (distribution_name, root_paths)) + _all_packages[distribution_name]["root_paths"] = root_paths + return MFExtModule(tl_package=tl_pkg, module=extension_module) return None -class _LazyLoader(object): - # This _LazyLoader implements the Importer Protocol defined in PEP 302 - # TODO: Need to move to find_spec, exec_module and create_module as - # find_module and load_module are deprecated +def _filter_files_package(package_name): + pkg = _all_packages.get(package_name) + if pkg and pkg["root_paths"] and pkg["meta_module"]: + meta_module = _attempt_load_module(pkg["meta_module"]) + if meta_module: + include_suffixes = meta_module.__dict__.get("include_suffixes") + exclude_suffixes = meta_module.__dict__.get("exclude_suffixes") + + # Behavior is as follows: + # - if nothing specified, include all files (so do nothing here) + # - if include_suffixes, only include those suffixes + # - if *not* include_suffixes but exclude_suffixes, include everything *except* + # files ending with that suffix + if include_suffixes: + new_files = [ + f + for f in pkg["files"] + if any([f.endswith(suffix) for suffix in include_suffixes]) + ] + elif exclude_suffixes: + new_files = [ + f + for f in pkg["files"] + if not any([f.endswith(suffix) for suffix in exclude_suffixes]) + ] + else: + new_files = pkg["files"] + pkg["files"] = new_files + + +def _filter_files_all(): + for p in _all_packages: + _filter_files_package(p) + + +class _AliasLoader(Loader): + def __init__(self, alias, orig): + self._alias = alias + self._orig = orig + + def create_module(self, spec): + _ext_debug( + "Loading aliased module '%s' at '%s' " % (str(self._orig), spec.name) + ) + if isinstance(self._orig, str): + try: + return importlib.import_module(self._orig) + except ImportError: + raise ImportError( + "No module found '%s' (aliasing '%s')" % (spec.name, self._orig) + ) + elif isinstance(self._orig, types.ModuleType): + # We are aliasing a module, so we just return that one + return self._orig + else: + return super().create_module(spec) + + def exec_module(self, module): + # Override the name to make it a bit nicer. We keep the old name so that + # we can refer to it when we load submodules + if not hasattr(module, "__orig_name__"): + module.__orig_name__ = module.__name__ + module.__name__ = self._alias + + +class _OrigLoader(Loader): + def __init__( + self, + fullname, + orig_loader, + previously_loaded_module=None, + previously_loaded_parent_module=None, + ): + self._fullname = fullname + self._orig_loader = orig_loader + self._previously_loaded_module = previously_loaded_module + self._previously_loaded_parent_module = previously_loaded_parent_module + + def create_module(self, spec): + _ext_debug( + "Loading original module '%s' (will be loaded at '%s'); spec is %s" + % (spec.name, self._fullname, str(spec)) + ) + self._orig_name = spec.name + return self._orig_loader.create_module(spec) + + def exec_module(self, module): + try: + # Perform all actions of the original loader + self._orig_loader.exec_module(module) + except BaseException: + raise # We re-raise it always; the `finally` clause will still restore things + else: + # It loaded, we move and rename appropriately + module.__spec__.name = self._fullname + module.__orig_name__ = module.__name__ + module.__name__ = self._fullname + module.__package__ = module.__spec__.parent # assumption since 3.6 + sys.modules[self._fullname] = module + del sys.modules[self._orig_name] + + finally: + # At this point, the original module is loaded with the original name. We + # want to replace it with previously_loaded_module if it exists. We + # also replace the parent properly + if self._previously_loaded_module: + sys.modules[self._orig_name] = self._previously_loaded_module + if self._previously_loaded_parent_module: + sys.modules[ + ".".join(self._orig_name.split(".")[:-1]) + ] = self._previously_loaded_parent_module + + +class _LazyFinder(MetaPathFinder): + # This _LazyFinder implements the Importer Protocol defined in PEP 302 def __init__(self, handled): - # Modules directly loaded (this is either new modules or overrides of existing ones) + # Dictionary: + # Key: name of the module to handle + # Value: + # - A string: a pathspec to the module to load + # - A module: the module to load self._handled = handled if handled else {} - # This is used to revert back to regular loading when trying to load + # This is used to revert to regular loading when trying to load # the over-ridden module - self._tempexcluded = set() - - # This is used when loading a module alias to load any submodule - self._alias_to_orig = {} - - def find_module(self, fullname, path=None): - if fullname in self._tempexcluded: + self._temp_excluded_prefix = set() + + # This is used to determine if we should be searching in _orig modules. Basically, + # when a relative import is done from a module in _orig, we want to search in + # the _orig "tree" + self._orig_search_paths = set() + + def find_spec(self, fullname, path, target=None): + # If we are trying to load a shadowed module (ending in ._orig), we don't + # say we handle it + _ext_debug( + "Looking for %s in %s with target %s" % (fullname, str(path), target) + ) + if any([fullname.startswith(e) for e in self._temp_excluded_prefix]): return None - if fullname in self._handled or ( - fullname.endswith("._orig") and fullname[:-6] in self._handled - ): - return self - name_parts = fullname.split(".") - if len(name_parts) > 1 and name_parts[-1] != "_orig": - # We check if we had an alias created for this module and if so, - # we are going to load it to properly fully create aliases all - # the way down. - parent_name = ".".join(name_parts[:-1]) - if parent_name in self._alias_to_orig: - return self - return None - def load_module(self, fullname): - if fullname in sys.modules: - return sys.modules[fullname] - if not self._can_handle_orig_module() and fullname.endswith("._orig"): - # We return a nicer error message - raise ImportError( - "Attempting to load '%s' -- loading shadowed modules in Metaflow " - "Extensions are only supported in Python 3.4+" % fullname + # If this is something we directly handle, return our loader + if fullname in self._handled: + return importlib.util.spec_from_loader( + fullname, _AliasLoader(fullname, self._handled[fullname]) ) - to_import = self._handled.get(fullname, None) - - # If to_import is None, two cases: - # - we are loading a ._orig module - # - OR we are loading a submodule - if to_import is None: - if fullname.endswith("._orig"): - try: - # We exclude this module temporarily from what we handle to - # revert back to the non-shadowing mode of import - self._tempexcluded.add(fullname) - to_import = importlib.util.find_spec(fullname) - finally: - self._tempexcluded.remove(fullname) - else: - name_parts = fullname.split(".") - submodule = name_parts[-1] - parent_name = ".".join(name_parts[:-1]) - to_import = ".".join([self._alias_to_orig[parent_name], submodule]) - if isinstance(to_import, str): - try: - to_import_mod = importlib.import_module(to_import) - except ImportError: - raise ImportError( - "No module found '%s' (aliasing %s)" % (fullname, to_import) + # For the first pass when we try to load a shadowed module, we send it back + # without the ._orig and that will find the original spec of the module + # Note that we handle mymodule._orig.orig_submodule as well as mymodule._orig. + # Basically, the original module and any of the original submodules are + # available under _orig. + name_parts = fullname.split(".") + try: + orig_idx = name_parts.index("_orig") + except ValueError: + orig_idx = -1 + if orig_idx > -1 and ".".join(name_parts[:orig_idx]) in self._handled: + orig_name = ".".join(name_parts[:orig_idx] + name_parts[orig_idx + 1 :]) + parent_name = None + if orig_idx != len(name_parts) - 1: + # We have a parent module under the _orig portion so for example, if + # we load mymodule._orig.orig_submodule, our parent is mymodule._orig. + # However, since mymodule is currently shadowed, we need to reset + # the parent module properly. We know it is already loaded (since modules + # are loaded hierarchically) + parent_name = ".".join( + name_parts[:orig_idx] + name_parts[orig_idx + 1 : -1] ) - sys.modules[fullname] = to_import_mod - self._alias_to_orig[fullname] = to_import_mod.__name__ - elif isinstance(to_import, types.ModuleType): - sys.modules[fullname] = to_import - self._alias_to_orig[fullname] = to_import.__name__ - elif self._can_handle_orig_module() and isinstance(to_import, ModuleSpec): - # This loads modules that end in _orig - m = importlib.util.module_from_spec(to_import) - to_import.loader.exec_module(m) - sys.modules[fullname] = m - elif to_import is None and fullname.endswith("._orig"): - # This happens when trying to access a shadowed ._orig module - # when actually, there is no shadowed module; print a nicer message - # Condition is a bit overkill and most likely only checking to_import - # would be OK. Being extra sure in case _LazyLoader is misused and - # a None value is passed in. - raise ImportError( - "Metaflow Extensions shadowed module '%s' does not exist" % fullname + _ext_debug("Looking for original module '%s'" % orig_name) + prefix = ".".join(name_parts[:orig_idx]) + self._temp_excluded_prefix.add(prefix) + # We also have to remove the module temporarily while we look for the + # new spec since otherwise it returns the spec of that loaded module. + # module is also restored *after* we call `create_module` in the loader + # otherwise it just returns None. We also swap out the parent module so that + # the search can start from there. + loaded_module = sys.modules.get(orig_name) + if loaded_module: + del sys.modules[orig_name] + parent_module = sys.modules.get(parent_name) if parent_name else None + if parent_module: + sys.modules[parent_name] = sys.modules[".".join([parent_name, "_orig"])] + + # This finds the spec that would have existed had we not added all our + # _LazyFinders + spec = importlib.util.find_spec(orig_name) + + self._temp_excluded_prefix.remove(prefix) + + if not spec: + return None + + if spec.submodule_search_locations: + self._orig_search_paths.update(spec.submodule_search_locations) + + _ext_debug("Found original spec %s" % spec) + + # Change the spec + spec.loader = _OrigLoader( + fullname, + spec.loader, + loaded_module, + parent_module, ) - else: - raise ImportError - return sys.modules[fullname] - @staticmethod - def _can_handle_orig_module(): - return sys.version_info[0] >= 3 and sys.version_info[1] >= 4 + return spec + + for p in path or []: + if p in self._orig_search_paths: + # We need to look in some of the "_orig" modules + orig_override_name = ".".join( + name_parts[:-1] + ["_orig", name_parts[-1]] + ) + _ext_debug( + "Looking for %s as an original module: searching for %s" + % (fullname, orig_override_name) + ) + return importlib.util.find_spec(orig_override_name) + if len(name_parts) > 1: + # This checks for submodules of things we handle. We check for the most + # specific submodule match and use that + chop_idx = 1 + while chop_idx < len(name_parts): + parent_name = ".".join(name_parts[:-chop_idx]) + if parent_name in self._handled: + orig = self._handled[parent_name] + if isinstance(orig, types.ModuleType): + orig_name = ".".join( + [orig.__orig_name__] + name_parts[-chop_idx:] + ) + else: + orig_name = ".".join([orig] + name_parts[-chop_idx:]) + return importlib.util.spec_from_loader( + fullname, _AliasLoader(fullname, orig_name) + ) + chop_idx += 1 + return None diff --git a/metaflow/flowspec.py b/metaflow/flowspec.py index fb9dec0ad76..606a400e94d 100644 --- a/metaflow/flowspec.py +++ b/metaflow/flowspec.py @@ -7,7 +7,7 @@ from types import FunctionType, MethodType from . import cmd_with_io -from .parameters import Parameter +from .parameters import DelayedEvaluationParameter, Parameter from .exception import ( MetaflowException, MissingInMergeArtifactsException, @@ -51,7 +51,6 @@ class FlowSpec(object): Attributes ---------- - script_name index input """ @@ -107,6 +106,8 @@ def __init__(self, use_cli=True): @property def script_name(self): """ + [Legacy function - do not use. Use `current` instead] + Returns the name of the script containing the flow Returns @@ -144,9 +145,8 @@ def _set_constants(self, graph, kwargs): for var, param in self._get_parameters(): seen.add(var) val = kwargs[param.name.replace("-", "_").lower()] - # Support for delayed evaluation of parameters. This is used for - # includefile in particular - if callable(val): + # Support for delayed evaluation of parameters. + if isinstance(val, DelayedEvaluationParameter): val = val() val = val.split(param.separator) if val and param.separator else val setattr(self, var, val) @@ -203,6 +203,8 @@ def _set_datastore(self, datastore): def __iter__(self): """ + [Legacy function - do not use] + Iterate over all steps in the Flow Returns @@ -223,26 +225,27 @@ def __getattr__(self, name): raise AttributeError("Flow %s has no attribute '%s'" % (self.name, name)) def cmd(self, cmdline, input={}, output=[]): + """ + [Legacy function - do not use] + """ return cmd_with_io.cmd(cmdline, input=input, output=output) @property def index(self): """ - Index of the task in a foreach step + The index of this foreach branch. In a foreach step, multiple instances of this step (tasks) will be executed, - one for each element in the foreach. - This property returns the zero based index of the current task. If this is not - a foreach step, this returns None. + one for each element in the foreach. This property returns the zero based index + of the current task. If this is not a foreach step, this returns None. - See Also - -------- - foreach_stack: A detailed example is given in the documentation of this function + If you need to know the indices of the parent tasks in a nested foreach, use + `FlowSpec.foreach_stack`. Returns ------- int - Index of the task in a foreach step + Index of the task in a foreach step. """ if self._foreach_stack: return self._foreach_stack[-1].index @@ -250,30 +253,28 @@ def index(self): @property def input(self): """ - Value passed to the task in a foreach step + The value of the foreach artifact in this foreach branch. In a foreach step, multiple instances of this step (tasks) will be executed, - one for each element in the foreach. - This property returns the element passed to the current task. If this is not - a foreach step, this returns None. + one for each element in the foreach. This property returns the element passed + to the current task. If this is not a foreach step, this returns None. - See Also - -------- - foreach_stack: A detailed example is given in the documentation of this function + If you need to know the values of the parent tasks in a nested foreach, use + `FlowSpec.foreach_stack`. Returns ------- object - Input passed to the task (can be any object) + Input passed to the foreach task. """ return self._find_input() def foreach_stack(self): """ - Returns the current stack of foreach steps for the current step + Returns the current stack of foreach indexes and values for the current step. - This effectively corresponds to the indexes and values at the various levels of nesting. - For example, considering the following code: + Use this information to understand what data is being processed in the current + foreach branch. For example, considering the following code: ``` @step def root(self): @@ -289,26 +290,31 @@ def nest_1(self): def nest_2(self): foo = self.foreach_stack() ``` - foo will take the following values in the various tasks for nest_2: + + `foo` will take the following values in the various tasks for nest_2: + ``` [(0, 3, 'a'), (0, 4, 'd')] [(0, 3, 'a'), (1, 4, 'e')] ... [(0, 3, 'a'), (3, 4, 'g')] [(1, 3, 'b'), (0, 4, 'd')] ... - + ``` where each tuple corresponds to: - - the index of the task for that level of the loop - - the number of splits for that level of the loop - - the value for that level of the loop + + - The index of the task for that level of the loop. + - The number of splits for that level of the loop. + - The value for that level of the loop. + Note that the last tuple returned in a task corresponds to: - - first element: value returned by self.index - - third element: value returned by self.input + + - 1st element: value returned by `self.index`. + - 3rd element: value returned by `self.input`. Returns ------- List[Tuple[int, int, object]] - An array describing the current stack of foreach steps + An array describing the current stack of foreach steps. """ return [ (frame.index, frame.num_splits, self._find_input(stack_index=i)) @@ -349,15 +355,16 @@ def _find_input(self, stack_index=None): def merge_artifacts(self, inputs, exclude=[], include=[]): """ - Merge the artifacts coming from each merge branch (from inputs) + Helper function for merging artifacts in a join step. This function takes all the artifacts coming from the branches of a join point and assigns them to self in the calling step. Only artifacts not set in the current step are considered. If, for a given artifact, different - values are present on the incoming edges, an error will be thrown (and the artifacts - that "conflict" will be reported). + values are present on the incoming edges, an error will be thrown and the artifacts + that conflict will be reported. As a few examples, in the simple graph: A splitting into B and C and joining in D: + ``` A: self.x = 5 self.y = 6 @@ -369,34 +376,34 @@ def merge_artifacts(self, inputs, exclude=[], include=[]): D: merge_artifacts(inputs) - + ``` In D, the following artifacts are set: - - y (value: 6), b_var (value: 1) - - if from_b and from_c are the same, x will be accessible and have value from_b - - if from_b and from_c are different, an error will be thrown. To prevent this error, - you need to manually set self.x in D to a merged value (for example the max) prior to - calling merge_artifacts. + - `y` (value: 6), `b_var` (value: 1) + - if `from_b` and `from_c` are the same, `x` will be accessible and have value `from_b` + - if `from_b` and `from_c` are different, an error will be thrown. To prevent this error, + you need to manually set `self.x` in D to a merged value (for example the max) prior to + calling `merge_artifacts`. Parameters ---------- inputs : List[Steps] - Incoming steps to the join point + Incoming steps to the join point. exclude : List[str], optional If specified, do not consider merging artifacts with a name in `exclude`. - Cannot specify if `include` is also specified + Cannot specify if `include` is also specified. include : List[str], optional If specified, only merge artifacts specified. Cannot specify if `exclude` is - also specified + also specified. Raises ------ MetaflowException - This exception is thrown if this is not called in a join step + This exception is thrown if this is not called in a join step. UnhandledInMergeArtifactsException - This exception is thrown in case of unresolved conflicts + This exception is thrown in case of unresolved conflicts. MissingInMergeArtifactsException - This exception is thrown in case an artifact specified in `include cannot - be found + This exception is thrown in case an artifact specified in `include` cannot + be found. """ node = self._graph[self._current_step] if node.type != "join": @@ -436,7 +443,7 @@ def merge_artifacts(self, inputs, exclude=[], include=[]): if v not in to_merge and not hasattr(self, v): missing.append(v) if unresolved: - # We have unresolved conflicts so we do not set anything and error out + # We have unresolved conflicts, so we do not set anything and error out msg = ( "Step *{step}* cannot merge the following artifacts due to them " "having conflicting values:\n[{artifacts}].\nTo remedy this issue, " @@ -483,22 +490,32 @@ def _validate_ubf_step(self, step_name): def next(self, *dsts, **kwargs): """ - Indicates the next step to execute at the end of this step + Indicates the next step to execute after this step has completed. - This statement should appear once and only once in each and every step (except the `end` - step). Furthermore, it should be the last statement in the step. + This statement should appear as the last statement of each step, except + the end step. There are several valid formats to specify the next step: - - Straight-line connection: self.next(self.next_step) where `next_step` is a method in - the current class decorated with the `@step` decorator - - Static fan-out connection: self.next(self.step1, self.step2, ...) where `stepX` are - methods in the current class decorated with the `@step` decorator - - Foreach branch: - self.next(self.foreach_step, foreach='foreach_iterator') - In this situation, `foreach_step` is a method in the current class decorated with the - `@step` docorator and `foreach_iterator` is a variable name in the current class that - evaluates to an iterator. A task will be launched for each value in the iterator and - each task will execute the code specified by the step `foreach_step`. + + - Straight-line connection: `self.next(self.next_step)` where `next_step` is a method in + the current class decorated with the `@step` decorator. + + - Static fan-out connection: `self.next(self.step1, self.step2, ...)` where `stepX` are + methods in the current class decorated with the `@step` decorator. + + - Foreach branch: + ``` + self.next(self.foreach_step, foreach='foreach_iterator') + ``` + In this situation, `foreach_step` is a method in the current class decorated with the + `@step` decorator and `foreach_iterator` is a variable name in the current class that + evaluates to an iterator. A task will be launched for each value in the iterator and + each task will execute the code specified by the step `foreach_step`. + + Parameters + ---------- + dsts : Method + One or more methods annotated with `@step`. Raises ------ diff --git a/metaflow/graph.py b/metaflow/graph.py index 9651f34bb2f..4ea41e0114f 100644 --- a/metaflow/graph.py +++ b/metaflow/graph.py @@ -5,7 +5,7 @@ def deindent_docstring(doc): if doc: - # Find the indent to remove from the doctring. We consider the following possibilities: + # Find the indent to remove from the docstring. We consider the following possibilities: # Option 1: # """This is the first line # This is the second line @@ -186,7 +186,7 @@ def _create_nodes(self, flow): def _postprocess(self): # any node who has a foreach as any of its split parents - # has is_inside_foreach=True *unless* all of those foreaches + # has is_inside_foreach=True *unless* all of those `foreach`s # are joined by the node for node in self.nodes.values(): foreaches = [ diff --git a/metaflow/includefile.py b/metaflow/includefile.py index 9131c012ed6..eac99560e42 100644 --- a/metaflow/includefile.py +++ b/metaflow/includefile.py @@ -1,3 +1,4 @@ +from collections import namedtuple import gzip import io @@ -8,259 +9,218 @@ from metaflow._vendor import click -from . import parameters -from .current import current from .exception import MetaflowException -from .metaflow_config import DATATOOLS_LOCALROOT, DATATOOLS_SUFFIX -from .parameters import DeployTimeField, Parameter -from .util import to_unicode +from .parameters import ( + DelayedEvaluationParameter, + DeployTimeField, + Parameter, + ParameterContext, +) +from .util import get_username -try: - # python2 - from urlparse import urlparse -except: - # python3 - from urllib.parse import urlparse +import functools -# TODO: This local "client" and the general notion of dataclients should probably -# be moved somewhere else. Putting here to keep this change compact for now -class MetaflowLocalURLException(MetaflowException): - headline = "Invalid path" +# _tracefunc_depth = 0 -class MetaflowLocalNotFound(MetaflowException): - headline = "Local object not found" +# def tracefunc(func): +# """Decorates a function to show its trace.""" +# @functools.wraps(func) +# def tracefunc_closure(*args, **kwargs): +# global _tracefunc_depth +# """The closure.""" +# print(f"{_tracefunc_depth}: {func.__name__}(args={args}, kwargs={kwargs})") +# _tracefunc_depth += 1 +# result = func(*args, **kwargs) +# _tracefunc_depth -= 1 +# print(f"{_tracefunc_depth} => {result}") +# return result -class LocalObject(object): - """ - This object represents a local object. It is a very thin wrapper - to allow it to be used in the same way as the S3Object (only as needed - in this usecase) - - Get or list calls return one or more of LocalObjects. - """ - - def __init__(self, url, path): - - # all fields of S3Object should return a unicode object - def ensure_unicode(x): - return None if x is None else to_unicode(x) - - path = ensure_unicode(path) - - self._path = path - self._url = url - - if self._path: - try: - os.stat(self._path) - except FileNotFoundError: - self._path = None - - @property - def exists(self): - """ - Does this key correspond to an actual file? - """ - return self._path is not None and os.path.isfile(self._path) - - @property - def url(self): - """ - Local location of the object; this is the path prefixed with local:// - """ - return self._url - - @property - def path(self): - """ - Path to the local file - """ - return self._path +# return tracefunc_closure -class Local(object): - """ - This class allows you to access the local filesystem in a way similar to the S3 datatools - client. It is a stripped down version for now and only implements the functionality needed - for this use case. - - In the future, we may want to allow it to be used in a way similar to the S3() client. - """ - - @staticmethod - def _makedirs(path): - try: - os.makedirs(path) - except OSError as x: - if x.errno == 17: - return - else: - raise - - @classmethod - def get_root_from_config(cls, echo, create_on_absent=True): - result = DATATOOLS_LOCALROOT - if result is None: - from .datastore.local_storage import LocalStorage - - result = LocalStorage.get_datastore_root_from_config(echo, create_on_absent) - result = os.path.join(result, DATATOOLS_SUFFIX) - if create_on_absent and not os.path.exists(result): - os.mkdir(result) - return result - - def __init__(self): - """ - Initialize a new context for Local file operations. This object is based used as - a context manager for a with statement. - """ - pass - - def __enter__(self): - return self - - def __exit__(self, *args): - pass - - def _path(self, key): - key = to_unicode(key) - if key.startswith(u"local://"): - return key[8:] - elif key[0] != u"/": - if current.is_running_flow: - raise MetaflowLocalURLException( - "Specify Local(run=self) when you use Local inside a running " - "flow. Otherwise you have to use Local with full " - "local:// urls or absolute paths." - ) - else: - raise MetaflowLocalURLException( - "Initialize Local with an 'localroot' or 'run' if you don't " - "want to specify full local:// urls or absolute paths." - ) - else: - return key - - def get(self, key=None, return_missing=False): - p = self._path(key) - url = u"local://%s" % p - if not os.path.isfile(p): - if return_missing: - p = None - else: - raise MetaflowLocalNotFound("Local URL %s not found" % url) - return LocalObject(url, p) - - def put(self, key, obj, overwrite=True): - p = self._path(key) - if overwrite or (not os.path.exists(p)): - Local._makedirs(os.path.dirname(p)) - with open(p, "wb") as f: - f.write(obj) - return u"local://%s" % p +_DelayedExecContext = namedtuple( + "_DelayedExecContext", "flow_name path is_text encoding handler_type echo" +) # From here on out, this is the IncludeFile implementation. -from .datatools import S3 +from metaflow.plugins.datatools import Local, S3 +from metaflow.plugins.azure.includefile_support import Azure +from metaflow.plugins.gcp.includefile_support import GS -DATACLIENTS = {"local": Local, "s3": S3} +DATACLIENTS = { + "local": Local, + "s3": S3, + "azure": Azure, + "gs": GS, +} -class LocalFile: - def __init__(self, is_text, encoding, path): - self._is_text = is_text - self._encoding = encoding - self._path = path +class IncludedFile(object): + # Thin wrapper to indicate to the MF client that this object is special + # and should be handled as an IncludedFile when returning it (ie: fetching + # the actual content) - @classmethod - def is_file_handled(cls, path): - if path: - decoded_value = Uploader.decode_value(to_unicode(path)) - if decoded_value["type"] == "self": - return ( - True, - LocalFile( - decoded_value["is_text"], - decoded_value["encoding"], - decoded_value["url"], - ), - None, - ) - path = decoded_value["url"] - for prefix, handler in DATACLIENTS.items(): - if path.startswith(u"%s://" % prefix): - return True, Uploader(handler), None - try: - with open(path, mode="r") as _: - pass - except OSError: - return False, None, "IncludeFile: could not open file '%s'" % path - return True, None, None + # @tracefunc + def __init__(self, descriptor): + self._descriptor = descriptor + self._cached_size = None - def __str__(self): - return self._path + @property + def descriptor(self): + return self._descriptor - def __repr__(self): - return self._path - - def __call__(self, ctx): - # We check again if this is a local file that exists. We do this here because - # we always convert local files to DeployTimeFields irrespective of whether - # the file exists. - ok, _, err = LocalFile.is_file_handled(self._path) - if not ok: - raise MetaflowException(err) - client = DATACLIENTS.get(ctx.ds_type) - if client: - return Uploader(client).store( - ctx.flow_name, self._path, self._is_text, self._encoding, ctx.logger + @property + # @tracefunc + def size(self): + if self._cached_size is not None: + return self._cached_size + handler = UPLOADERS.get(self.descriptor.get("type", None), None) + if handler is None: + raise MetaflowException( + "Could not interpret size of IncludedFile: %s" + % json.dumps(self.descriptor) ) - raise MetaflowException( - "IncludeFile: no client found for datastore type %s" % ctx.ds_type - ) + self._cached_size = handler.size(self._descriptor) + return self._cached_size + + # @tracefunc + def decode(self, name, var_type="Artifact"): + # We look for the uploader for it and decode it + handler = UPLOADERS.get(self.descriptor.get("type", None), None) + if handler is None: + raise MetaflowException( + "%s '%s' could not be loaded (IncludedFile) because no handler found: %s" + % (var_type, name, json.dumps(self.descriptor)) + ) + return handler.load(self._descriptor) class FilePathClass(click.ParamType): name = "FilePath" - # The logic for this class is as follows: - # - It will always return a path that indicates the persisted path of the file. - # + If the value is already such a string, nothing happens and it returns that same value - # + If the value is a LocalFile, it will persist the local file and return the path - # of the persisted file - # - The artifact will be persisted prior to any run (for non-scheduled runs through persist_constants) - # + This will therefore persist a simple string - # - When the parameter is loaded again, the load_parameter in the IncludeFile class will get called - # which will download and return the bytes of the persisted file. + def __init__(self, is_text, encoding): self._is_text = is_text self._encoding = encoding def convert(self, value, param, ctx): - if callable(value): - # Already a correct type + # Click can call convert multiple times, so we need to make sure to only + # convert once. This function will return a DelayedEvaluationParameter + # (if it needs to still perform an upload) or an IncludedFile if not + if isinstance(value, (DelayedEvaluationParameter, IncludedFile)): return value - value = os.path.expanduser(value) - ok, file_type, err = LocalFile.is_file_handled(value) - if not ok: - self.fail(err) - if file_type is None: - # Here, we need to store the file - return lambda is_text=self._is_text, encoding=self._encoding, value=value, ctx=parameters.context_proto: LocalFile( - is_text, encoding, value - )( - ctx + # Value will be a string containing one of two things: + # - Scenario A: a JSON blob indicating that the file has already been uploaded. + # This scenario this happens in is as follows: + # + `step-functions create` is called and the IncludeFile has a default + # value. At the time of creation, the file is uploaded and a URL is + # returned; this URL is packaged in a blob by Uploader and passed to + # step-functions as the value of the parameter. + # + when the step function actually runs, the value is passed to click + # through METAFLOW_INIT_XXX; this value is the one returned above + # - Scenario B: A path. The path can either be: + # + B.1: :// like s3://foo/bar or local:///foo/bar + # (right now, we are disabling support for this because the artifact + # can change unlike all other artifacts. It is trivial to re-enable + # + B.2: an actual path to a local file like /foo/bar + # In the first case, we just store an *external* reference to it (so we + # won't upload anything). In the second case we will want to upload something, + # but we only do that in the DelayedEvaluationParameter step. + + # ctx can be one of two things: + # - the click context (when called normally) + # - the ParameterContext (when called through _eval_default) + # If not a ParameterContext, we convert it to that + if not isinstance(ctx, ParameterContext): + ctx = ParameterContext( + flow_name=ctx.obj.flow.name, + user_name=get_username(), + parameter_name=param.name, + logger=ctx.obj.echo, + ds_type=ctx.obj.datastore_impl.TYPE, ) - elif isinstance(file_type, LocalFile): - # This is a default file that we evaluate now (to delay upload - # until *after* the flow is checked) - return lambda f=file_type, ctx=parameters.context_proto: f(ctx) + + if len(value) > 0 and value[0] == "{": + # This is a blob; no URL starts with `{`. We are thus in scenario A + try: + value = json.loads(value) + except json.JSONDecodeError as e: + raise MetaflowException( + "IncludeFile '%s' (value: %s) is malformed" % (param.name, value) + ) + # All processing has already been done, so we just convert to an `IncludedFile` + return IncludedFile(value) + + path = os.path.expanduser(value) + + prefix_pos = path.find("://") + if prefix_pos > 0: + # Scenario B.1 + raise MetaflowException( + "IncludeFile using a direct reference to a file in cloud storage is no " + "longer supported. Contact the Metaflow team if you need this supported" + ) + # if DATACLIENTS.get(path[:prefix_pos]) is None: + # self.fail( + # "IncludeFile: no handler for external file of type '%s' " + # "(given path is '%s')" % (path[:prefix_pos], path) + # ) + # # We don't need to do anything more -- the file is already uploaded so we + # # just return a blob indicating how to get the file. + # return IncludedFile( + # CURRENT_UPLOADER.encode_url( + # "external", path, is_text=self._is_text, encoding=self._encoding + # ) + # ) else: - # We will just store the URL in the datastore along with text/encoding info - return lambda is_text=self._is_text, encoding=self._encoding, value=value: Uploader.encode_url( - "external", value, is_text=is_text, encoding=encoding + # Scenario B.2 + # Check if this is a valid local file + try: + with open(path, mode="r") as _: + pass + except OSError: + self.fail("IncludeFile: could not open file '%s' for reading" % path) + handler = DATACLIENTS.get(ctx.ds_type) + if handler is None: + self.fail( + "IncludeFile: no data-client for datastore of type '%s'" + % ctx.ds_type + ) + + # Now that we have done preliminary checks, we will delay uploading it + # until later (so it happens after PyLint checks the flow, but we prepare + # everything for it) + lambda_ctx = _DelayedExecContext( + flow_name=ctx.flow_name, + path=path, + is_text=self._is_text, + encoding=self._encoding, + handler_type=ctx.ds_type, + echo=ctx.logger, + ) + + def _delayed_eval_func(ctx=lambda_ctx, return_str=False): + incl_file = IncludedFile( + CURRENT_UPLOADER.store( + ctx.flow_name, + ctx.path, + ctx.is_text, + ctx.encoding, + DATACLIENTS[ctx.handler_type], + ctx.echo, + ) + ) + if return_str: + return json.dumps(incl_file.descriptor) + return incl_file + + return DelayedEvaluationParameter( + ctx.parameter_name, + "default", + functools.partial(_delayed_eval_func, ctx=lambda_ctx), ) def __str__(self): @@ -271,84 +231,115 @@ def __repr__(self): class IncludeFile(Parameter): + """ + Includes a local file as a parameter for the flow. + + `IncludeFile` behaves like `Parameter` except that it reads its value from a file instead of + the command line. The user provides a path to a file on the command line. The file contents + are saved as a read-only artifact which is available in all steps of the flow. + + Parameters + ---------- + name : str + User-visible parameter name. + default : str or a function + Default path to a local file. A function + implies that the parameter corresponds to a *deploy-time parameter*. + is_text : bool + Convert the file contents to a string using the provided `encoding` (default: True). + If False, the artifact is stored in `bytes`. + encoding : str + Use this encoding to decode the file contexts if `is_text=True` (default: `utf-8`). + required : bool + Require that the user specified a value for the parameter. + `required=True` implies that the `default` is not used. + help : str + Help text to show in `run --help`. + show_default : bool + If True, show the default value in the help text (default: True). + """ + def __init__( self, name, required=False, is_text=True, encoding=None, help=None, **kwargs ): - # Defaults are DeployTimeField + # If a default is specified, it needs to be uploaded when the flow is deployed + # (for example when doing a `step-functions create`) so we make the default + # be a DeployTimeField. This means that it will be evaluated in two cases: + # - by deploy_time_eval for `step-functions create` and related. + # - by Click when evaluating the parameter. + # + # In the first case, we will need to fully upload the file whereas in the + # second case, we can just return the string as the FilePath.convert method + # will take care of evaluating things. v = kwargs.get("default") if v is not None: - _, file_type, _ = LocalFile.is_file_handled(v) - # Ignore error because we may never use the default - if file_type is None: - o = {"type": "self", "is_text": is_text, "encoding": encoding, "url": v} - kwargs["default"] = DeployTimeField( - name, - str, - "default", - lambda ctx, full_evaluation, o=o: LocalFile( - o["is_text"], o["encoding"], o["url"] - )(ctx) - if full_evaluation - else json.dumps(o), - print_representation=v, - ) - else: - kwargs["default"] = DeployTimeField( - name, - str, - "default", - lambda _, __, is_text=is_text, encoding=encoding, v=v: Uploader.encode_url( - "external-default", v, is_text=is_text, encoding=encoding - ), - print_representation=v, - ) + # If the default is a callable, we have two DeployTimeField: + # - the callable nature of the default will require us to "call" the default + # (so that is the outer DeployTimeField) + # - IncludeFile defaults are always DeployTimeFields (since they need to be + # uploaded) + # + # Therefore, if the default value is itself a callable, we will have + # a DeployTimeField (upload the file) wrapping another DeployTimeField + # (call the default) + if callable(v) and not isinstance(v, DeployTimeField): + # If default is a callable, make it a DeployTimeField (the inner one) + v = DeployTimeField(name, str, "default", v, return_str=True) + kwargs["default"] = DeployTimeField( + name, + str, + "default", + IncludeFile._eval_default(is_text, encoding, v), + print_representation=v, + ) super(IncludeFile, self).__init__( name, required=required, help=help, type=FilePathClass(is_text, encoding), - **kwargs + **kwargs, ) - def load_parameter(self, val): - if val is None: - return val - ok, file_type, err = LocalFile.is_file_handled(val) - if not ok: - raise MetaflowException( - "Parameter '%s' could not be loaded: %s" % (self.name, err) - ) - if file_type is None or isinstance(file_type, LocalFile): - raise MetaflowException( - "Parameter '%s' was not properly converted" % self.name - ) - return file_type.load(val) + def load_parameter(self, v): + if v is None: + return v + return v.decode(self.name, var_type="Parameter") + @staticmethod + def _eval_default(is_text, encoding, default_path): + # NOTE: If changing name of this function, check comments that refer to it to + # update it. + def do_eval(ctx, deploy_time): + if isinstance(default_path, DeployTimeField): + d = default_path(deploy_time=deploy_time) + else: + d = default_path + if deploy_time: + fp = FilePathClass(is_text, encoding) + val = fp.convert(d, None, ctx) + if isinstance(val, DelayedEvaluationParameter): + val = val() + # At this point this is an IncludedFile, but we need to make it + # into a string so that it can be properly saved. + return json.dumps(val.descriptor) + else: + return d -class Uploader: + return do_eval - file_type = "uploader-v1" - def __init__(self, client_class): - self._client_class = client_class +class UploaderV1: + file_type = "uploader-v1" - @staticmethod - def encode_url(url_type, url, **kwargs): - # Avoid encoding twice (default -> URL -> _convert method of FilePath for example) - if url is None or len(url) == 0 or url[0] == "{": - return url + @classmethod + def encode_url(cls, url_type, url, **kwargs): return_value = {"type": url_type, "url": url} return_value.update(kwargs) - return json.dumps(return_value) - - @staticmethod - def decode_value(value): - if value is None or len(value) == 0 or value[0] != "{": - return {"type": "base", "url": value} - return json.loads(value) + return return_value - def store(self, flow_name, path, is_text, encoding, echo): + @classmethod + def store(cls, flow_name, path, is_text, encoding, handler, echo): sz = os.path.getsize(path) unit = ["B", "KB", "MB", "GB", "TB"] pos = 0 @@ -370,39 +361,147 @@ def store(self, flow_name, path, is_text, encoding, echo): "large to be properly handled by Python 2.7" % path ) sha = sha1(input_file).hexdigest() - path = os.path.join( - self._client_class.get_root_from_config(echo, True), flow_name, sha - ) + path = os.path.join(handler.get_root_from_config(echo, True), flow_name, sha) buf = io.BytesIO() + with gzip.GzipFile(fileobj=buf, mode="wb", compresslevel=3) as f: f.write(input_file) buf.seek(0) - with self._client_class() as client: + + with handler() as client: url = client.put(path, buf.getvalue(), overwrite=False) - echo("File persisted at %s" % url) - return Uploader.encode_url( - Uploader.file_type, url, is_text=is_text, encoding=encoding + + return cls.encode_url(cls.file_type, url, is_text=is_text, encoding=encoding) + + @classmethod + def size(cls, descriptor): + # We never have the size so we look it up + url = descriptor["url"] + handler = cls._get_handler(url) + with handler() as client: + obj = client.info(url, return_missing=True) + if obj.exists: + return obj.size + raise FileNotFoundError("File at '%s' does not exist" % url) + + @classmethod + def load(cls, descriptor): + url = descriptor["url"] + handler = cls._get_handler(url) + with handler() as client: + obj = client.get(url, return_missing=True) + if obj.exists: + if descriptor["type"] == cls.file_type: + # We saved this file directly, so we know how to read it out + with gzip.GzipFile(filename=obj.path, mode="rb") as f: + if descriptor["is_text"]: + return io.TextIOWrapper( + f, encoding=descriptor.get("encoding") + ).read() + return f.read() + else: + # We open this file according to the is_text and encoding information + if descriptor["is_text"]: + return io.open( + obj.path, mode="rt", encoding=descriptor.get("encoding") + ).read() + else: + return io.open(obj.path, mode="rb").read() + raise FileNotFoundError("File at '%s' does not exist" % descriptor["url"]) + + @staticmethod + def _get_handler(url): + prefix_pos = url.find("://") + if prefix_pos < 0: + raise MetaflowException("Malformed URL: '%s'" % url) + prefix = url[:prefix_pos] + handler = DATACLIENTS.get(prefix) + if handler is None: + raise MetaflowException("Could not find data client for '%s'" % prefix) + return handler + + +class UploaderV2: + + file_type = "uploader-v2" + + @classmethod + def encode_url(cls, url_type, url, **kwargs): + return_value = { + "note": "Internal representation of IncludeFile(%s)" % url, + "type": cls.file_type, + "sub-type": url_type, + "url": url, + } + return_value.update(kwargs) + return return_value + + @classmethod + def store(cls, flow_name, path, is_text, encoding, handler, echo): + r = UploaderV1.store(flow_name, path, is_text, encoding, handler, echo) + + # In V2, we store size for faster access + r["note"] = "Internal representation of IncludeFile(%s)" % path + r["type"] = cls.file_type + r["sub-type"] = "uploaded" + r["size"] = os.stat(path).st_size + return r + + @classmethod + def size(cls, descriptor): + if descriptor["sub-type"] == "uploaded": + return descriptor["size"] + else: + # This was a file that was external, so we get information on it + url = descriptor["url"] + handler = cls._get_handler(url) + with handler() as client: + obj = client.info(url, return_missing=True) + if obj.exists: + return obj.size + raise FileNotFoundError( + "%s file at '%s' does not exist" + % (descriptor["sub-type"].capitalize(), url) ) - def load(self, value): - value_info = Uploader.decode_value(value) - with self._client_class() as client: - obj = client.get(value_info["url"], return_missing=True) + @classmethod + def load(cls, descriptor): + url = descriptor["url"] + # We know the URL is in a :// format so we just extract the handler + handler = cls._get_handler(url) + with handler() as client: + obj = client.get(url, return_missing=True) if obj.exists: - if value_info["type"] == Uploader.file_type: - # We saved this file directly so we know how to read it out + if descriptor["sub-type"] == "uploaded": + # We saved this file directly, so we know how to read it out with gzip.GzipFile(filename=obj.path, mode="rb") as f: - if value_info["is_text"]: + if descriptor["is_text"]: return io.TextIOWrapper( - f, encoding=value_info.get("encoding") + f, encoding=descriptor.get("encoding") ).read() return f.read() else: # We open this file according to the is_text and encoding information - if value_info["is_text"]: + if descriptor["is_text"]: return io.open( - obj.path, mode="rt", encoding=value_info.get("encoding") + obj.path, mode="rt", encoding=descriptor.get("encoding") ).read() else: return io.open(obj.path, mode="rb").read() - raise FileNotFoundError("File at %s does not exist" % value_info["url"]) + # If we are here, the file does not exist + raise FileNotFoundError( + "%s file at '%s' does not exist" + % (descriptor["sub-type"].capitalize(), url) + ) + + @staticmethod + def _get_handler(url): + return UploaderV1._get_handler(url) + + +UPLOADERS = { + "uploader-v1": UploaderV1, + "external": UploaderV1, + "uploader-v2": UploaderV2, +} +CURRENT_UPLOADER = UploaderV2 diff --git a/metaflow/lint.py b/metaflow/lint.py index c282cc31b76..ebcb8e98e24 100644 --- a/metaflow/lint.py +++ b/metaflow/lint.py @@ -246,6 +246,7 @@ def traverse(node, split_stack): ) else: raise LintWarn(msg2.format(node), node.func_lineno) + # check that incoming steps come from the same lineage # (no cross joins) def parents(n): diff --git a/metaflow/metadata/heartbeat.py b/metaflow/metadata/heartbeat.py index d2f01f2fb4b..7a73c0cf90d 100644 --- a/metaflow/metadata/heartbeat.py +++ b/metaflow/metadata/heartbeat.py @@ -3,8 +3,8 @@ import json from threading import Thread -from metaflow.sidecar_messages import MessageTypes, Message -from metaflow.metaflow_config import METADATA_SERVICE_HEADERS +from metaflow.sidecar import MessageTypes, Message +from metaflow.metaflow_config import SERVICE_HEADERS from metaflow.exception import MetaflowException HB_URL_KEY = "hb_url" @@ -19,8 +19,8 @@ def __init__(self, msg): class MetadataHeartBeat(object): def __init__(self): - self.headers = METADATA_SERVICE_HEADERS - self.req_thread = Thread(target=self.ping) + self.headers = SERVICE_HEADERS + self.req_thread = Thread(target=self._ping) self.req_thread.daemon = True self.default_frequency_secs = 10 self.hb_url = None @@ -28,19 +28,22 @@ def __init__(self): def process_message(self, msg): # type: (Message) -> None if msg.msg_type == MessageTypes.SHUTDOWN: - # todo shutdown doesnt do anything yet? should it still be called - self.shutdown() - if (not self.req_thread.is_alive()) and msg.msg_type == MessageTypes.LOG_EVENT: + self._shutdown() + if not self.req_thread.is_alive(): # set post url self.hb_url = msg.payload[HB_URL_KEY] # start thread self.req_thread.start() - def ping(self): + @classmethod + def get_worker(cls): + return cls + + def _ping(self): retry_counter = 0 while True: try: - frequency_secs = self.heartbeat() + frequency_secs = self._heartbeat() if frequency_secs is None or frequency_secs <= 0: frequency_secs = self.default_frequency_secs @@ -49,9 +52,9 @@ def ping(self): retry_counter = 0 except HeartBeatException as e: retry_counter = retry_counter + 1 - time.sleep(4 ** retry_counter) + time.sleep(4**retry_counter) - def heartbeat(self): + def _heartbeat(self): if self.hb_url is not None: response = requests.post(url=self.hb_url, data="{}", headers=self.headers) # Unfortunately, response.json() returns a string that we need @@ -67,6 +70,6 @@ def heartbeat(self): ) return None - def shutdown(self): + def _shutdown(self): # attempts sending one last heartbeat - self.heartbeat() + self._heartbeat() diff --git a/metaflow/metadata/metadata.py b/metaflow/metadata/metadata.py index c20b441f09e..70427c3589a 100644 --- a/metaflow/metadata/metadata.py +++ b/metaflow/metadata/metadata.py @@ -3,11 +3,11 @@ import re import time from collections import namedtuple -from datetime import datetime - -from metaflow.exception import MetaflowInternalError -from metaflow.util import get_username, resolve_identity +from itertools import chain +from metaflow.exception import MetaflowInternalError, MetaflowTaggingError +from metaflow.tagging_util import validate_tag +from metaflow.util import get_username, resolve_identity_as_tuple, is_stringish DataArtifact = namedtuple("DataArtifact", "name ds_type ds_root url type sha") @@ -47,6 +47,33 @@ def decorator(cls): return decorator +class ObjectOrder: + # Consider this list a constant that should never change. + # Lots of code depend on the membership of this list as + # well as exact ordering + _order_as_list = [ + "root", + "flow", + "run", + "step", + "task", + "artifact", + "metadata", + "self", + ] + _order_as_dict = {v: i for i, v in enumerate(_order_as_list)} + + @staticmethod + def order_to_type(order): + if order < len(ObjectOrder._order_as_list): + return ObjectOrder._order_as_list[order] + return None + + @staticmethod + def type_to_order(obj_type): + return ObjectOrder._order_as_dict.get(obj_type) + + @with_metaclass(MetadataProviderMeta) class MetadataProvider(object): @classmethod @@ -128,6 +155,10 @@ def register_run_id(self, run_id, tags=None, sys_tags=None): Tags to apply to this particular run, by default None sys_tags : list, optional System tags to apply to this particular run, by default None + Returns + ------- + bool + True if a new run was registered; False if it already existed """ raise NotImplementedError() @@ -173,6 +204,10 @@ def register_task_id( Tags to apply to this particular run, by default [] sys_tags : list, optional System tags to apply to this particular run, by default [] + Returns + ------- + bool + True if a new run was registered; False if it already existed """ raise NotImplementedError() @@ -355,22 +390,12 @@ def get_object(cls, obj_type, sub_type, filters, attempt, *args): object or list : Depending on the call, the type of object return varies """ - obj_order = { - "root": 0, - "flow": 1, - "run": 2, - "step": 3, - "task": 4, - "artifact": 5, - "metadata": 6, - "self": 7, - } - type_order = obj_order.get(obj_type) - sub_order = obj_order.get(sub_type) + type_order = ObjectOrder.type_to_order(obj_type) + sub_order = ObjectOrder.type_to_order(sub_type) if type_order is None: raise MetaflowInternalError(msg="Cannot find type %s" % obj_type) - if type_order > 5: + if type_order >= ObjectOrder.type_to_order("metadata"): raise MetaflowInternalError(msg="Type %s is not allowed" % obj_type) if sub_order is None: @@ -400,17 +425,93 @@ def get_object(cls, obj_type, sub_type, filters, attempt, *args): pre_filter = cls._get_object_internal( obj_type, type_order, sub_type, sub_order, filters, attempt_int, *args ) - if attempt_int is None or sub_order != 6: + if attempt_int is None or sub_type != "metadata": # If no attempt or not for metadata, just return as is return pre_filter return MetadataProvider._reconstruct_metadata_for_attempt( pre_filter, attempt_int ) + @classmethod + def mutate_user_tags_for_run( + cls, flow_id, run_id, tags_to_remove=None, tags_to_add=None + ): + """ + Mutate the set of user tags for a run. + + Removals logically get applied after additions. Operations occur as a batch atomically. + Parameters + ---------- + flow_id : str + Flow id, that the run belongs to. + run_id: str + Run id, together with flow_id, that identifies the specific Run whose tags to mutate + tags_to_remove: iterable over str + Iterable over tags to remove + tags_to_add: iterable over str + Iterable over tags to add + + Return + ------ + Run tags after mutation operations + """ + # perform common validation, across all provider implementations + if tags_to_remove is None: + tags_to_remove = [] + if tags_to_add is None: + tags_to_add = [] + if not tags_to_add and not tags_to_remove: + raise MetaflowTaggingError("Must add or remove at least one tag") + + if is_stringish(tags_to_add): + raise MetaflowTaggingError("tags_to_add may not be a string") + + if is_stringish(tags_to_remove): + raise MetaflowTaggingError("tags_to_remove may not be a string") + + def _is_iterable(something): + try: + iter(something) + return True + except TypeError: + return False + + if not _is_iterable(tags_to_add): + raise MetaflowTaggingError("tags_to_add must be iterable") + if not _is_iterable(tags_to_remove): + raise MetaflowTaggingError("tags_to_remove must be iterable") + + # check each tag is valid + for tag in chain(tags_to_add, tags_to_remove): + validate_tag(tag) + + # onto subclass implementation + final_user_tags = cls._mutate_user_tags_for_run( + flow_id, run_id, tags_to_add=tags_to_add, tags_to_remove=tags_to_remove + ) + return final_user_tags + + @classmethod + def _mutate_user_tags_for_run( + cls, flow_id, run_id, tags_to_add=None, tags_to_remove=None + ): + """ + To be implemented by subclasses of MetadataProvider. + + See mutate_user_tags_for_run() for expectations. + """ + raise NotImplementedError() + def _all_obj_elements(self, tags=None, sys_tags=None): + return MetadataProvider._all_obj_elements_static( + self._flow_name, tags=tags, sys_tags=sys_tags + ) + + @staticmethod + def _all_obj_elements_static(flow_name, tags=None, sys_tags=None): user = get_username() return { - "flow_id": self._flow_name, + "flow_id": flow_name, "user_name": user, "tags": list(tags) if tags else [], "system_tags": list(sys_tags) if sys_tags else [], @@ -424,11 +525,17 @@ def _flow_to_json(self): return {"flow_id": self._flow_name, "ts_epoch": int(round(time.time() * 1000))} def _run_to_json(self, run_id=None, tags=None, sys_tags=None): + return MetadataProvider._run_to_json_static( + self._flow_name, run_id=run_id, tags=tags, sys_tags=sys_tags + ) + + @staticmethod + def _run_to_json_static(flow_name, run_id=None, tags=None, sys_tags=None): if run_id is not None: d = {"run_number": run_id} else: d = {} - d.update(self._all_obj_elements(tags, sys_tags)) + d.update(MetadataProvider._all_obj_elements_static(flow_name, tags, sys_tags)) return d def _step_to_json(self, run_id, step_name, tags=None, sys_tags=None): @@ -497,28 +604,55 @@ def _metadata_to_json(self, run_id, step_name, task_id, metadata): for datum in metadata ] - def _tags(self): + def _get_system_info_as_dict(self): + """This function drives: + - sticky system tags initialization + - task-level metadata generation + """ + sys_info = dict() env = self._environment.get_environment_info() - tags = [ - resolve_identity(), - "runtime:" + env["runtime"], - "python_version:" + env["python_version_code"], - "date:" + datetime.utcnow().strftime("%Y-%m-%d"), - ] + sys_info["runtime"] = env["runtime"] + sys_info["python_version"] = env["python_version_code"] + identity_type, identity_value = resolve_identity_as_tuple() + sys_info[identity_type] = identity_value if env["metaflow_version"]: - tags.append("metaflow_version:" + env["metaflow_version"]) + sys_info["metaflow_version"] = env["metaflow_version"] if "metaflow_r_version" in env: - tags.append("metaflow_r_version:" + env["metaflow_r_version"]) + sys_info["metaflow_r_version"] = env["metaflow_r_version"] if "r_version_code" in env: - tags.append("r_version:" + env["r_version_code"]) - return tags + sys_info["r_version"] = env["r_version_code"] + return sys_info + + def _get_system_tags(self): + """Convert system info dictionary into a list of system tags""" + return [ + "{}:{}".format(k, v) for k, v in self._get_system_info_as_dict().items() + ] - def _register_code_package_metadata(self, run_id, step_name, task_id, attempt): + def _register_system_metadata(self, run_id, step_name, task_id, attempt): + """Gather up system and code packaging info and register them as task metadata""" metadata = [] + # Take everything from system info and store them as metadata + sys_info = self._get_system_info_as_dict() + + # field, and type could get long in theory...can the metadata backend handle it? + # E.g. as of 5/9/2022 Metadata service's DB says VARCHAR(255). + # It is likely overkill to fail a flow over an over-flow. We should expect the + # backend to try to tolerate this (e.g. enlarge columns, truncation fallback). + metadata.extend( + MetaDatum( + field=str(k), + value=str(v), + type=str(k), + tags=["attempt_id:{0}".format(attempt)], + ) + for k, v in sys_info.items() + ) + # Also store code packaging information code_sha = os.environ.get("METAFLOW_CODE_SHA") - code_url = os.environ.get("METAFLOW_CODE_URL") - code_ds = os.environ.get("METAFLOW_CODE_DS") if code_sha: + code_url = os.environ.get("METAFLOW_CODE_URL") + code_ds = os.environ.get("METAFLOW_CODE_DS") metadata.append( MetaDatum( field="code-package", @@ -529,8 +663,6 @@ def _register_code_package_metadata(self, run_id, step_name, task_id, attempt): tags=["attempt_id:{0}".format(attempt)], ) ) - # We don't tag with attempt_id here because not readily available; this - # is ok though as this doesn't change from attempt to attempt. if metadata: self.register_metadata(run_id, step_name, task_id, metadata) @@ -604,4 +736,4 @@ def __init__(self, environment, flow, event_logger, monitor): self._monitor = monitor self._environment = environment self._runtime = os.environ.get("METAFLOW_RUNTIME_NAME", "dev") - self.add_sticky_tags(sys_tags=self._tags()) + self.add_sticky_tags(sys_tags=self._get_system_tags()) diff --git a/metaflow/metadata/util.py b/metaflow/metadata/util.py index ec6ab58fcd5..705cfb50f52 100644 --- a/metaflow/metadata/util.py +++ b/metaflow/metadata/util.py @@ -5,7 +5,7 @@ from distutils.dir_util import copy_tree from metaflow import util -from metaflow.datastore.local_storage import LocalStorage +from metaflow.plugins.datastores.local_storage import LocalStorage def sync_local_metadata_to_datastore(metadata_local_dir, task_ds): @@ -27,7 +27,7 @@ def echo_none(*args, **kwargs): _, tarball = next(task_ds.parent_datastore.load_data([key_to_load])) with util.TempDir() as td: with tarfile.open(fileobj=BytesIO(tarball), mode="r:gz") as tar: - tar.extractall(td) + util.tar_safe_extract(tar, td) copy_tree( os.path.join(td, metadata_local_dir), LocalStorage.get_datastore_root_from_config(echo_none), diff --git a/metaflow/metaflow_config.py b/metaflow/metaflow_config.py index 8a2037bf465..8bc78adee99 100644 --- a/metaflow/metaflow_config.py +++ b/metaflow/metaflow_config.py @@ -1,55 +1,34 @@ -import os -import json import logging -import pkg_resources +import os import sys import types +import pkg_resources from metaflow.exception import MetaflowException +from metaflow.metaflow_config_funcs import from_conf, get_validate_choice_fn # Disable multithreading security on MacOS if sys.platform == "darwin": os.environ["OBJC_DISABLE_INITIALIZE_FORK_SAFETY"] = "YES" - -def init_config(): - # Read configuration from $METAFLOW_HOME/config_.json. - home = os.environ.get("METAFLOW_HOME", "~/.metaflowconfig") - profile = os.environ.get("METAFLOW_PROFILE") - path_to_config = os.path.join(home, "config.json") - if profile: - path_to_config = os.path.join(home, "config_%s.json" % profile) - path_to_config = os.path.expanduser(path_to_config) - config = {} - if os.path.exists(path_to_config): - with open(path_to_config) as f: - return json.load(f) - elif profile: - raise MetaflowException( - "Unable to locate METAFLOW_PROFILE '%s' in '%s')" % (profile, home) - ) - return config - - -# Initialize defaults required to setup environment variables. -METAFLOW_CONFIG = init_config() - - -def from_conf(name, default=None): - return os.environ.get(name, METAFLOW_CONFIG.get(name, default)) - +## NOTE: Just like Click's auto_envar_prefix `METAFLOW` (see in cli.py), all environment +## variables here are also named METAFLOW_XXX. So, for example, in the statement: +## `DEFAULT_DATASTORE = from_conf("DEFAULT_DATASTORE", "local")`, to override the default +## value, either set `METAFLOW_DEFAULT_DATASTORE` in your configuration file or set +## an environment variable called `METAFLOW_DEFAULT_DATASTORE` ### # Default configuration ### -DEFAULT_DATASTORE = from_conf("METAFLOW_DEFAULT_DATASTORE", "local") -DEFAULT_ENVIRONMENT = from_conf("METAFLOW_DEFAULT_ENVIRONMENT", "local") -DEFAULT_EVENT_LOGGER = from_conf("METAFLOW_DEFAULT_EVENT_LOGGER", "nullSidecarLogger") -DEFAULT_METADATA = from_conf("METAFLOW_DEFAULT_METADATA", "local") -DEFAULT_MONITOR = from_conf("METAFLOW_DEFAULT_MONITOR", "nullSidecarMonitor") -DEFAULT_PACKAGE_SUFFIXES = from_conf("METAFLOW_DEFAULT_PACKAGE_SUFFIXES", ".py,.R,.RDS") -DEFAULT_AWS_CLIENT_PROVIDER = from_conf("METAFLOW_DEFAULT_AWS_CLIENT_PROVIDER", "boto3") + +DEFAULT_DATASTORE = from_conf("DEFAULT_DATASTORE", "local") +DEFAULT_ENVIRONMENT = from_conf("DEFAULT_ENVIRONMENT", "local") +DEFAULT_EVENT_LOGGER = from_conf("DEFAULT_EVENT_LOGGER", "nullSidecarLogger") +DEFAULT_METADATA = from_conf("DEFAULT_METADATA", "local") +DEFAULT_MONITOR = from_conf("DEFAULT_MONITOR", "nullSidecarMonitor") +DEFAULT_PACKAGE_SUFFIXES = from_conf("DEFAULT_PACKAGE_SUFFIXES", ".py,.R,.RDS") +DEFAULT_AWS_CLIENT_PROVIDER = from_conf("DEFAULT_AWS_CLIENT_PROVIDER", "boto3") ### @@ -57,159 +36,250 @@ def from_conf(name, default=None): ### # Path to the local directory to store artifacts for 'local' datastore. DATASTORE_LOCAL_DIR = ".metaflow" -DATASTORE_SYSROOT_LOCAL = from_conf("METAFLOW_DATASTORE_SYSROOT_LOCAL") +DATASTORE_SYSROOT_LOCAL = from_conf("DATASTORE_SYSROOT_LOCAL") # S3 bucket and prefix to store artifacts for 's3' datastore. -DATASTORE_SYSROOT_S3 = from_conf("METAFLOW_DATASTORE_SYSROOT_S3") +DATASTORE_SYSROOT_S3 = from_conf("DATASTORE_SYSROOT_S3") +# Azure Blob Storage container and blob prefix +DATASTORE_SYSROOT_AZURE = from_conf("DATASTORE_SYSROOT_AZURE") +DATASTORE_SYSROOT_GS = from_conf("DATASTORE_SYSROOT_GS") +# GS bucket and prefix to store artifacts for 'gs' datastore + + +### +# Datastore local cache +### + +# Path to the client cache +CLIENT_CACHE_PATH = from_conf("CLIENT_CACHE_PATH", "/tmp/metaflow_client") +# Maximum size (in bytes) of the cache +CLIENT_CACHE_MAX_SIZE = from_conf("CLIENT_CACHE_MAX_SIZE", 10000) +# Maximum number of cached Flow and TaskDatastores in the cache +CLIENT_CACHE_MAX_FLOWDATASTORE_COUNT = from_conf( + "CLIENT_CACHE_MAX_FLOWDATASTORE_COUNT", 50 +) +CLIENT_CACHE_MAX_TASKDATASTORE_COUNT = from_conf( + "CLIENT_CACHE_MAX_TASKDATASTORE_COUNT", CLIENT_CACHE_MAX_FLOWDATASTORE_COUNT * 100 +) + + +### +# Datatools (S3) configuration +### +S3_ENDPOINT_URL = from_conf("S3_ENDPOINT_URL") +S3_VERIFY_CERTIFICATE = from_conf("S3_VERIFY_CERTIFICATE") + +# S3 retry configuration +# This is useful if you want to "fail fast" on S3 operations; use with caution +# though as this may increase failures. Note that this is the number of *retries* +# so setting it to 0 means each operation will be tried once. +S3_RETRY_COUNT = from_conf("S3_RETRY_COUNT", 7) + +# Number of retries on *transient* failures (such as SlowDown errors). Note +# that if after S3_TRANSIENT_RETRY_COUNT times, all operations haven't been done, +# it will try up to S3_RETRY_COUNT again so the total number of tries can be up to +# (S3_RETRY_COUNT + 1) * (S3_TRANSIENT_RETRY_COUNT + 1) +# You typically want this number fairly high as transient retires are "cheap" (only +# operations that have not succeeded retry as opposed to all operations for the +# top-level retries) +S3_TRANSIENT_RETRY_COUNT = from_conf("S3_TRANSIENT_RETRY_COUNT", 20) + +# Threshold to start printing warnings for an AWS retry +RETRY_WARNING_THRESHOLD = 3 + # S3 datatools root location -DATATOOLS_SUFFIX = from_conf("METAFLOW_DATATOOLS_SUFFIX", "data") +DATATOOLS_SUFFIX = from_conf("DATATOOLS_SUFFIX", "data") DATATOOLS_S3ROOT = from_conf( - "METAFLOW_DATATOOLS_S3ROOT", - os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATATOOLS_SUFFIX) - if from_conf("METAFLOW_DATASTORE_SYSROOT_S3") + "DATATOOLS_S3ROOT", + os.path.join(DATASTORE_SYSROOT_S3, DATATOOLS_SUFFIX) + if DATASTORE_SYSROOT_S3 + else None, +) + +DATATOOLS_CLIENT_PARAMS = from_conf("DATATOOLS_CLIENT_PARAMS", {}) +if S3_ENDPOINT_URL: + DATATOOLS_CLIENT_PARAMS["endpoint_url"] = S3_ENDPOINT_URL +if S3_VERIFY_CERTIFICATE: + DATATOOLS_CLIENT_PARAMS["verify"] = S3_VERIFY_CERTIFICATE + +DATATOOLS_SESSION_VARS = from_conf("DATATOOLS_SESSION_VARS", {}) + +# Azure datatools root location +# Note: we do not expose an actual datatools library for Azure (like we do for S3) +# Similar to DATATOOLS_LOCALROOT, this is used ONLY by the IncludeFile's internal implementation. +DATATOOLS_AZUREROOT = from_conf( + "DATATOOLS_AZUREROOT", + os.path.join(DATASTORE_SYSROOT_AZURE, DATATOOLS_SUFFIX) + if DATASTORE_SYSROOT_AZURE + else None, +) +# GS datatools root location +# Note: we do not expose an actual datatools library for GS (like we do for S3) +# Similar to DATATOOLS_LOCALROOT, this is used ONLY by the IncludeFile's internal implementation. +DATATOOLS_GSROOT = from_conf( + "DATATOOLS_GSROOT", + os.path.join(DATASTORE_SYSROOT_GS, DATATOOLS_SUFFIX) + if DATASTORE_SYSROOT_GS else None, ) # Local datatools root location DATATOOLS_LOCALROOT = from_conf( - "METAFLOW_DATATOOLS_LOCALROOT", - os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_LOCAL"), DATATOOLS_SUFFIX) - if from_conf("METAFLOW_DATASTORE_SYSROOT_LOCAL") + "DATATOOLS_LOCALROOT", + os.path.join(DATASTORE_SYSROOT_LOCAL, DATATOOLS_SUFFIX) + if DATASTORE_SYSROOT_LOCAL else None, ) +# The root directory to save artifact pulls in, when using S3 or Azure +ARTIFACT_LOCALROOT = from_conf("ARTIFACT_LOCALROOT", os.getcwd()) + # Cards related config variables -DATASTORE_CARD_SUFFIX = "mf.cards" -DATASTORE_CARD_LOCALROOT = from_conf("METAFLOW_CARD_LOCALROOT") -DATASTORE_CARD_S3ROOT = from_conf( - "METAFLOW_CARD_S3ROOT", - os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATASTORE_CARD_SUFFIX) - if from_conf("METAFLOW_DATASTORE_SYSROOT_S3") +CARD_SUFFIX = "mf.cards" +CARD_LOCALROOT = from_conf("CARD_LOCALROOT") +CARD_S3ROOT = from_conf( + "CARD_S3ROOT", + os.path.join(DATASTORE_SYSROOT_S3, CARD_SUFFIX) if DATASTORE_SYSROOT_S3 else None, +) +CARD_AZUREROOT = from_conf( + "CARD_AZUREROOT", + os.path.join(DATASTORE_SYSROOT_AZURE, CARD_SUFFIX) + if DATASTORE_SYSROOT_AZURE else None, ) -CARD_NO_WARNING = from_conf("METAFLOW_CARD_NO_WARNING", False) +CARD_GSROOT = from_conf( + "CARD_GSROOT", + os.path.join(DATASTORE_SYSROOT_GS, CARD_SUFFIX) if DATASTORE_SYSROOT_GS else None, +) +CARD_NO_WARNING = from_conf("CARD_NO_WARNING", False) -# S3 endpoint url -S3_ENDPOINT_URL = from_conf("METAFLOW_S3_ENDPOINT_URL", None) -S3_VERIFY_CERTIFICATE = from_conf("METAFLOW_S3_VERIFY_CERTIFICATE", None) +SKIP_CARD_DUALWRITE = from_conf("SKIP_CARD_DUALWRITE", False) -# S3 retry configuration -# This is useful if you want to "fail fast" on S3 operations; use with caution -# though as this may increase failures. Note that this is the number of *retries* -# so setting it to 0 means each operation will be tried once. -S3_RETRY_COUNT = int(from_conf("METAFLOW_S3_RETRY_COUNT", 7)) +# Azure storage account URL +AZURE_STORAGE_BLOB_SERVICE_ENDPOINT = from_conf("AZURE_STORAGE_BLOB_SERVICE_ENDPOINT") -### -# Datastore local cache -### -# Path to the client cache -CLIENT_CACHE_PATH = from_conf("METAFLOW_CLIENT_CACHE_PATH", "/tmp/metaflow_client") -# Maximum size (in bytes) of the cache -CLIENT_CACHE_MAX_SIZE = int(from_conf("METAFLOW_CLIENT_CACHE_MAX_SIZE", 10000)) -# Maximum number of cached Flow and TaskDatastores in the cache -CLIENT_CACHE_MAX_FLOWDATASTORE_COUNT = int( - from_conf("METAFLOW_CLIENT_CACHE_MAX_FLOWDATASTORE_COUNT", 50) -) -CLIENT_CACHE_MAX_TASKDATASTORE_COUNT = int( - from_conf( - "METAFLOW_CLIENT_CACHE_MAX_TASKDATASTORE_COUNT", - CLIENT_CACHE_MAX_FLOWDATASTORE_COUNT * 100, - ) +# Azure storage can use process-based parallelism instead of threads. +# Processes perform better for high throughput workloads (e.g. many huge artifacts) +AZURE_STORAGE_WORKLOAD_TYPE = from_conf( + "AZURE_STORAGE_WORKLOAD_TYPE", + default="general", + validate_fn=get_validate_choice_fn(["general", "high_throughput"]), ) +# GS storage can use process-based parallelism instead of threads. +# Processes perform better for high throughput workloads (e.g. many huge artifacts) +GS_STORAGE_WORKLOAD_TYPE = from_conf( + "GS_STORAGE_WORKLOAD_TYPE", + "general", + validate_fn=get_validate_choice_fn(["general", "high_throughput"]), +) ### # Metadata configuration ### -METADATA_SERVICE_URL = from_conf("METAFLOW_SERVICE_URL") -METADATA_SERVICE_NUM_RETRIES = int(from_conf("METAFLOW_SERVICE_RETRY_COUNT", 5)) -METADATA_SERVICE_AUTH_KEY = from_conf("METAFLOW_SERVICE_AUTH_KEY") -METADATA_SERVICE_HEADERS = json.loads(from_conf("METAFLOW_SERVICE_HEADERS", "{}")) -if METADATA_SERVICE_AUTH_KEY is not None: - METADATA_SERVICE_HEADERS["x-api-key"] = METADATA_SERVICE_AUTH_KEY +SERVICE_URL = from_conf("SERVICE_URL") +SERVICE_RETRY_COUNT = from_conf("SERVICE_RETRY_COUNT", 5) +SERVICE_AUTH_KEY = from_conf("SERVICE_AUTH_KEY") +SERVICE_HEADERS = from_conf("SERVICE_HEADERS", {}) +if SERVICE_AUTH_KEY is not None: + SERVICE_HEADERS["x-api-key"] = SERVICE_AUTH_KEY +# Checks version compatibility with Metadata service +SERVICE_VERSION_CHECK = from_conf("SERVICE_VERSION_CHECK", True) # Default container image -DEFAULT_CONTAINER_IMAGE = from_conf("METAFLOW_DEFAULT_CONTAINER_IMAGE") +DEFAULT_CONTAINER_IMAGE = from_conf("DEFAULT_CONTAINER_IMAGE") # Default container registry -DEFAULT_CONTAINER_REGISTRY = from_conf("METAFLOW_DEFAULT_CONTAINER_REGISTRY") +DEFAULT_CONTAINER_REGISTRY = from_conf("DEFAULT_CONTAINER_REGISTRY") ### # AWS Batch configuration ### # IAM role for AWS Batch container with Amazon S3 access # (and AWS DynamoDb access for AWS StepFunctions, if enabled) -ECS_S3_ACCESS_IAM_ROLE = from_conf("METAFLOW_ECS_S3_ACCESS_IAM_ROLE") +ECS_S3_ACCESS_IAM_ROLE = from_conf("ECS_S3_ACCESS_IAM_ROLE") # IAM role for AWS Batch container for AWS Fargate -ECS_FARGATE_EXECUTION_ROLE = from_conf("METAFLOW_ECS_FARGATE_EXECUTION_ROLE") +ECS_FARGATE_EXECUTION_ROLE = from_conf("ECS_FARGATE_EXECUTION_ROLE") # Job queue for AWS Batch -BATCH_JOB_QUEUE = from_conf("METAFLOW_BATCH_JOB_QUEUE") +BATCH_JOB_QUEUE = from_conf("BATCH_JOB_QUEUE") # Default container image for AWS Batch -BATCH_CONTAINER_IMAGE = ( - from_conf("METAFLOW_BATCH_CONTAINER_IMAGE") or DEFAULT_CONTAINER_IMAGE -) +BATCH_CONTAINER_IMAGE = from_conf("BATCH_CONTAINER_IMAGE", DEFAULT_CONTAINER_IMAGE) # Default container registry for AWS Batch -BATCH_CONTAINER_REGISTRY = ( - from_conf("METAFLOW_BATCH_CONTAINER_REGISTRY") or DEFAULT_CONTAINER_REGISTRY +BATCH_CONTAINER_REGISTRY = from_conf( + "BATCH_CONTAINER_REGISTRY", DEFAULT_CONTAINER_REGISTRY ) # Metadata service URL for AWS Batch -BATCH_METADATA_SERVICE_URL = from_conf( - "METAFLOW_SERVICE_INTERNAL_URL", METADATA_SERVICE_URL -) -BATCH_METADATA_SERVICE_HEADERS = METADATA_SERVICE_HEADERS +SERVICE_INTERNAL_URL = from_conf("SERVICE_INTERNAL_URL", SERVICE_URL) # Assign resource tags to AWS Batch jobs. Set to False by default since # it requires `Batch:TagResource` permissions which may not be available # in all Metaflow deployments. Hopefully, some day we can flip the # default to True. -BATCH_EMIT_TAGS = from_conf("METAFLOW_BATCH_EMIT_TAGS", False) +BATCH_EMIT_TAGS = from_conf("BATCH_EMIT_TAGS", False) ### # AWS Step Functions configuration ### # IAM role for AWS Step Functions with AWS Batch and AWS DynamoDb access # https://docs.aws.amazon.com/step-functions/latest/dg/batch-iam.html -SFN_IAM_ROLE = from_conf("METAFLOW_SFN_IAM_ROLE") +SFN_IAM_ROLE = from_conf("SFN_IAM_ROLE") # AWS DynamoDb Table name (with partition key - `pathspec` of type string) -SFN_DYNAMO_DB_TABLE = from_conf("METAFLOW_SFN_DYNAMO_DB_TABLE") +SFN_DYNAMO_DB_TABLE = from_conf("SFN_DYNAMO_DB_TABLE") # IAM role for AWS Events with AWS Step Functions access # https://docs.aws.amazon.com/eventbridge/latest/userguide/auth-and-access-control-eventbridge.html -EVENTS_SFN_ACCESS_IAM_ROLE = from_conf("METAFLOW_EVENTS_SFN_ACCESS_IAM_ROLE") +EVENTS_SFN_ACCESS_IAM_ROLE = from_conf("EVENTS_SFN_ACCESS_IAM_ROLE") # Prefix for AWS Step Functions state machines. Set to stack name for Metaflow # sandbox. -SFN_STATE_MACHINE_PREFIX = from_conf("METAFLOW_SFN_STATE_MACHINE_PREFIX") +SFN_STATE_MACHINE_PREFIX = from_conf("SFN_STATE_MACHINE_PREFIX") # Optional AWS CloudWatch Log Group ARN for emitting AWS Step Functions state # machine execution logs. This needs to be available when using the # `step-functions create --log-execution-history` command. -SFN_EXECUTION_LOG_GROUP_ARN = from_conf("METAFLOW_SFN_EXECUTION_LOG_GROUP_ARN") +SFN_EXECUTION_LOG_GROUP_ARN = from_conf("SFN_EXECUTION_LOG_GROUP_ARN") ### # Kubernetes configuration ### # Kubernetes namespace to use for all objects created by Metaflow -KUBERNETES_NAMESPACE = from_conf("METAFLOW_KUBERNETES_NAMESPACE", "default") -# Service account to use by K8S jobs created by Metaflow -KUBERNETES_SERVICE_ACCOUNT = from_conf("METAFLOW_KUBERNETES_SERVICE_ACCOUNT") +KUBERNETES_NAMESPACE = from_conf("KUBERNETES_NAMESPACE", "default") +# Default service account to use by K8S jobs created by Metaflow +KUBERNETES_SERVICE_ACCOUNT = from_conf("KUBERNETES_SERVICE_ACCOUNT") +# Default node selectors to use by K8S jobs created by Metaflow - foo=bar,baz=bab +KUBERNETES_NODE_SELECTOR = from_conf("KUBERNETES_NODE_SELECTOR", "") +KUBERNETES_TOLERATIONS = from_conf("KUBERNETES_TOLERATIONS", "") +KUBERNETES_SECRETS = from_conf("KUBERNETES_SECRETS", "") +# Default GPU vendor to use by K8S jobs created by Metaflow (supports nvidia, amd) +KUBERNETES_GPU_VENDOR = from_conf("KUBERNETES_GPU_VENDOR", "nvidia") # Default container image for K8S -KUBERNETES_CONTAINER_IMAGE = ( - from_conf("METAFLOW_KUBERNETES_CONTAINER_IMAGE") or DEFAULT_CONTAINER_IMAGE +KUBERNETES_CONTAINER_IMAGE = from_conf( + "KUBERNETES_CONTAINER_IMAGE", DEFAULT_CONTAINER_IMAGE ) # Default container registry for K8S -KUBERNETES_CONTAINER_REGISTRY = ( - from_conf("METAFLOW_KUBERNETES_CONTAINER_REGISTRY") or DEFAULT_CONTAINER_REGISTRY +KUBERNETES_CONTAINER_REGISTRY = from_conf( + "KUBERNETES_CONTAINER_REGISTRY", DEFAULT_CONTAINER_REGISTRY ) -# + +## +# Airflow Configuration +## +# This configuration sets `startup_timeout_seconds` in airflow's KubernetesPodOperator. +AIRFLOW_KUBERNETES_STARTUP_TIMEOUT_SECONDS = from_conf( + "AIRFLOW_KUBERNETES_STARTUP_TIMEOUT_SECONDS", 60 * 60 +) +# This configuration sets `kubernetes_conn_id` in airflow's KubernetesPodOperator. +AIRFLOW_KUBERNETES_CONN_ID = from_conf("AIRFLOW_KUBERNETES_CONN_ID") + ### # Conda configuration ### # Conda package root location on S3 -CONDA_PACKAGE_S3ROOT = from_conf( - "METAFLOW_CONDA_PACKAGE_S3ROOT", - "%s/conda" % from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), -) +CONDA_PACKAGE_S3ROOT = from_conf("CONDA_PACKAGE_S3ROOT") +# Conda package root location on Azure +CONDA_PACKAGE_AZUREROOT = from_conf("CONDA_PACKAGE_AZUREROOT") +# Conda package root location on GS +CONDA_PACKAGE_GSROOT = from_conf("CONDA_PACKAGE_GSROOT") # Use an alternate dependency resolver for conda packages instead of conda # Mamba promises faster package dependency resolution times, which # should result in an appreciable speedup in flow environment initialization. -CONDA_DEPENDENCY_RESOLVER = from_conf("METAFLOW_CONDA_DEPENDENCY_RESOLVER", "conda") +CONDA_DEPENDENCY_RESOLVER = from_conf("CONDA_DEPENDENCY_RESOLVER", "conda") ### # Debug configuration @@ -217,34 +287,31 @@ def from_conf(name, default=None): DEBUG_OPTIONS = ["subcommand", "sidecar", "s3client"] for typ in DEBUG_OPTIONS: - vars()["METAFLOW_DEBUG_%s" % typ.upper()] = from_conf( - "METAFLOW_DEBUG_%s" % typ.upper() - ) + vars()["DEBUG_%s" % typ.upper()] = from_conf("DEBUG_%s" % typ.upper()) ### # AWS Sandbox configuration ### # Boolean flag for metaflow AWS sandbox access -AWS_SANDBOX_ENABLED = bool(from_conf("METAFLOW_AWS_SANDBOX_ENABLED", False)) +AWS_SANDBOX_ENABLED = from_conf("AWS_SANDBOX_ENABLED", False) # Metaflow AWS sandbox auth endpoint -AWS_SANDBOX_STS_ENDPOINT_URL = from_conf("METAFLOW_SERVICE_URL") +AWS_SANDBOX_STS_ENDPOINT_URL = SERVICE_URL # Metaflow AWS sandbox API auth key -AWS_SANDBOX_API_KEY = from_conf("METAFLOW_AWS_SANDBOX_API_KEY") +AWS_SANDBOX_API_KEY = from_conf("AWS_SANDBOX_API_KEY") # Internal Metadata URL -AWS_SANDBOX_INTERNAL_SERVICE_URL = from_conf( - "METAFLOW_AWS_SANDBOX_INTERNAL_SERVICE_URL" -) +AWS_SANDBOX_INTERNAL_SERVICE_URL = from_conf("AWS_SANDBOX_INTERNAL_SERVICE_URL") # AWS region -AWS_SANDBOX_REGION = from_conf("METAFLOW_AWS_SANDBOX_REGION") +AWS_SANDBOX_REGION = from_conf("AWS_SANDBOX_REGION") # Finalize configuration if AWS_SANDBOX_ENABLED: os.environ["AWS_DEFAULT_REGION"] = AWS_SANDBOX_REGION - BATCH_METADATA_SERVICE_URL = AWS_SANDBOX_INTERNAL_SERVICE_URL - METADATA_SERVICE_HEADERS["x-api-key"] = AWS_SANDBOX_API_KEY - SFN_STATE_MACHINE_PREFIX = from_conf("METAFLOW_AWS_SANDBOX_STACK_NAME") + SERVICE_INTERNAL_URL = AWS_SANDBOX_INTERNAL_SERVICE_URL + SERVICE_HEADERS["x-api-key"] = AWS_SANDBOX_API_KEY + SFN_STATE_MACHINE_PREFIX = from_conf("AWS_SANDBOX_STACK_NAME") +KUBERNETES_SANDBOX_INIT_SCRIPT = from_conf("KUBERNETES_SANDBOX_INIT_SCRIPT") # MAX_ATTEMPTS is the maximum number of attempts, including the first # task, retries, and the final fallback task and its retries. @@ -279,15 +346,27 @@ def get_version(pkg): # PINNED_CONDA_LIBS are the libraries that metaflow depends on for execution # and are needed within a conda environment -def get_pinned_conda_libs(python_version): - return { +def get_pinned_conda_libs(python_version, datastore_type): + pins = { "requests": ">=2.21.0", - "boto3": ">=1.14.0", } + if datastore_type == "s3": + pins["boto3"] = ">=1.14.0" + elif datastore_type == "azure": + pins["azure-identity"] = ">=1.10.0" + pins["azure-storage-blob"] = ">=12.12.0" + elif datastore_type == "gs": + pins["google-cloud-storage"] = ">=2.5.0" + pins["google-auth"] = ">=2.11.0" + elif datastore_type == "local": + pass + else: + raise MetaflowException( + msg="conda lib pins for datastore %s are undefined" % (datastore_type,) + ) + return pins -METAFLOW_EXTENSIONS_ADDL_SUFFIXES = set([]) - # Check if there are extensions to Metaflow to load and override everything try: from metaflow.extension_support import get_modules @@ -300,19 +379,40 @@ def get_pinned_conda_libs(python_version): if n == "DEBUG_OPTIONS": DEBUG_OPTIONS.extend(o) for typ in o: - vars()["METAFLOW_DEBUG_%s" % typ.upper()] = from_conf( - "METAFLOW_DEBUG_%s" % typ.upper() + vars()["DEBUG_%s" % typ.upper()] = from_conf( + "DEBUG_%s" % typ.upper() ) - elif n == "METAFLOW_EXTENSIONS_ADDL_SUFFIXES": - METAFLOW_EXTENSIONS_ADDL_SUFFIXES.update(o) + elif n == "get_pinned_conda_libs": + + def _new_get_pinned_conda_libs( + python_version, datastore_type, f1=globals()[n], f2=o + ): + d1 = f1(python_version, datastore_type) + d2 = f2(python_version, datastore_type) + for k, v in d2.items(): + d1[k] = v if k not in d1 else ",".join([d1[k], v]) + return d1 + + globals()[n] = _new_get_pinned_conda_libs elif not n.startswith("__") and not isinstance(o, types.ModuleType): globals()[n] = o - METAFLOW_EXTENSIONS_ADDL_SUFFIXES = list(METAFLOW_EXTENSIONS_ADDL_SUFFIXES) - if len(METAFLOW_EXTENSIONS_ADDL_SUFFIXES) == 0: - METAFLOW_EXTENSIONS_ADDL_SUFFIXES = None finally: # Erase all temporary names to avoid leaking things - for _n in ["m", "n", "o", "ext_modules", "get_modules"]: + for _n in [ + "m", + "n", + "o", + "type", + "ext_modules", + "get_modules", + "_new_get_pinned_conda_libs", + "d1", + "d2", + "k", + "v", + "f1", + "f2", + ]: try: del globals()[_n] except KeyError: diff --git a/metaflow/metaflow_config_funcs.py b/metaflow/metaflow_config_funcs.py new file mode 100644 index 00000000000..365a18d75eb --- /dev/null +++ b/metaflow/metaflow_config_funcs.py @@ -0,0 +1,120 @@ +import json +import os + +from collections import namedtuple + +from metaflow.exception import MetaflowException +from metaflow.util import is_stringish + +ConfigValue = namedtuple("ConfigValue", "value serializer is_default") + +NON_CHANGED_VALUES = 1 +NULL_VALUES = 2 +ALL_VALUES = 3 + + +def init_config(): + # Read configuration from $METAFLOW_HOME/config_.json. + home = os.environ.get("METAFLOW_HOME", "~/.metaflowconfig") + profile = os.environ.get("METAFLOW_PROFILE") + path_to_config = os.path.join(home, "config.json") + if profile: + path_to_config = os.path.join(home, "config_%s.json" % profile) + path_to_config = os.path.expanduser(path_to_config) + config = {} + if os.path.exists(path_to_config): + with open(path_to_config, encoding="utf-8") as f: + return json.load(f) + elif profile: + raise MetaflowException( + "Unable to locate METAFLOW_PROFILE '%s' in '%s')" % (profile, home) + ) + return config + + +# Initialize defaults required to setup environment variables. +METAFLOW_CONFIG = init_config() + +_all_configs = {} + + +def config_values(include=0): + # By default, we just return non-null values and that + # are not default. This is the common use case because in all other cases, the code + # is sufficient to recreate the value (ie: there is no external source for the value) + for name, config_value in _all_configs.items(): + if (config_value.value is not None or include & NULL_VALUES) and ( + not config_value.is_default or include & NON_CHANGED_VALUES + ): + yield name, config_value.serializer(config_value.value) + + +def from_conf(name, default=None, validate_fn=None): + """ + First try to pull value from environment, then from metaflow config JSON + + Prior to a value being returned, we will validate using validate_fn (if provided). + Only non-None values are validated. + + validate_fn should accept (name, value). + If the value validates, return None, else raise an MetaflowException. + """ + env_name = "METAFLOW_%s" % name + is_default = True + value = os.environ.get(env_name, METAFLOW_CONFIG.get(env_name, default)) + if validate_fn and value is not None: + validate_fn(env_name, value) + if default is not None: + # In this case, value is definitely not None because default is the ultimate + # fallback and all other cases will return a string (even if an empty string) + if isinstance(default, (list, dict)): + # If we used the default, value is already a list or dict, else it is a + # string so we can just compare types to determine is_default + if isinstance(value, (list, dict)): + is_default = True + else: + try: + value = json.loads(value) + except json.JSONDecodeError: + raise ValueError( + "Expected a valid JSON for %s, got: %s" % (env_name, value) + ) + _all_configs[env_name] = ConfigValue( + value=value, + serializer=json.dumps, + is_default=is_default, + ) + return value + elif isinstance(default, (bool, int, float)) or is_stringish(default): + try: + value = type(default)(value) + # Here we can compare values + is_default = value == default + except ValueError: + raise ValueError( + "Expected a %s for %s, got: %s" % (type(default), env_name, value) + ) + else: + raise RuntimeError( + "Default of type %s for %s is not supported" % (type(default), env_name) + ) + _all_configs[env_name] = ConfigValue( + value=value, + serializer=str, + is_default=is_default, + ) + return value + + +def get_validate_choice_fn(choices): + """Returns a validate_fn for use with from_conf(). + The validate_fn will check a value against a list of allowed choices. + """ + + def _validate_choice(name, value): + if value not in choices: + raise MetaflowException( + "%s must be set to one of %s. Got '%s'." % (name, choices, value) + ) + + return _validate_choice diff --git a/metaflow/metaflow_environment.py b/metaflow/metaflow_environment.py index 476a02bf132..839e29a8640 100644 --- a/metaflow/metaflow_environment.py +++ b/metaflow/metaflow_environment.py @@ -28,7 +28,7 @@ def init_environment(self, echo): """ pass - def validate_environment(self, echo): + def validate_environment(self, echo, datastore_type): """ Run before any command to validate that we are operating in a desired environment. @@ -42,7 +42,7 @@ def decospecs(self): """ return () - def bootstrap_commands(self, step_name): + def bootstrap_commands(self, step_name, datastore_type): """ A list of shell commands to bootstrap this environment in a remote runtime. """ @@ -79,20 +79,80 @@ def get_client_info(cls, flow_name, metadata): """ return "Local environment" - def get_package_commands(self, code_package_url): + def _get_download_code_package_cmd(self, code_package_url, datastore_type): + """Return a command that downloads the code package from the datastore. We use various + cloud storage CLI tools because we don't have access to Metaflow codebase (which we + are about to download in the command). + + The command should download the package to "job.tar" in the current directory. + + It should work silently if everything goes well. + """ + if datastore_type == "s3": + return ( + '%s -m awscli ${METAFLOW_S3_ENDPOINT_URL:+--endpoint-url=\\"${METAFLOW_S3_ENDPOINT_URL}\\"} ' + + "s3 cp %s job.tar >/dev/null" + ) % (self._python(), code_package_url) + elif datastore_type == "azure": + from .plugins.azure.azure_utils import parse_azure_full_path + + container_name, blob = parse_azure_full_path(code_package_url) + # remove a trailing slash, if present + blob_endpoint = "${METAFLOW_AZURE_STORAGE_BLOB_SERVICE_ENDPOINT%/}" + return "download-azure-blob --blob-endpoint={blob_endpoint} --container={container} --blob={blob} --output-file=job.tar".format( + blob_endpoint=blob_endpoint, + blob=blob, + container=container_name, + ) + elif datastore_type == "gs": + from .plugins.gcp.gs_utils import parse_gs_full_path + + bucket_name, gs_object = parse_gs_full_path(code_package_url) + return ( + "download-gcp-object --bucket=%s --object=%s --output-file=job.tar" + % (bucket_name, gs_object) + ) + else: + raise NotImplementedError( + "We don't know how to generate a download code package cmd for datastore %s" + % datastore_type + ) + + def _get_install_dependencies_cmd(self, datastore_type): + cmds = ["%s -m pip install requests -qqq" % self._python()] + if datastore_type == "s3": + cmds.append("%s -m pip install awscli boto3 -qqq" % self._python()) + elif datastore_type == "azure": + cmds.append( + "%s -m pip install azure-identity azure-storage-blob simple-azure-blob-downloader -qqq" + % self._python() + ) + elif datastore_type == "gs": + cmds.append( + "%s -m pip install google-cloud-storage google-auth simple-gcp-object-downloader -qqq" + % self._python() + ) + else: + raise NotImplementedError( + "We don't know how to generate an install dependencies cmd for datastore %s" + % datastore_type + ) + return " && ".join(cmds) + + def get_package_commands(self, code_package_url, datastore_type): cmds = [ BASH_MFLOG, "mflog 'Setting up task environment.'", - "%s -m pip install awscli requests boto3 -qqq" % self._python(), + self._get_install_dependencies_cmd(datastore_type), "mkdir metaflow", "cd metaflow", "mkdir .metaflow", # mute local datastore creation log "i=0; while [ $i -le 5 ]; do " "mflog 'Downloading code package...'; " - "%s -m awscli s3 cp %s job.tar >/dev/null && \ - mflog 'Code package downloaded.' && break; " + + self._get_download_code_package_cmd(code_package_url, datastore_type) + + " && mflog 'Code package downloaded.' && break; " "sleep 10; i=$((i+1)); " - "done" % (self._python(), code_package_url), + "done", "if [ $i -gt 5 ]; then " "mflog 'Failed to download code package from %s " "after 6 tries. Exiting...' && exit 1; " diff --git a/metaflow/metaflow_version.py b/metaflow/metaflow_version.py index 108b666779f..9a36dc79ae0 100644 --- a/metaflow/metaflow_version.py +++ b/metaflow/metaflow_version.py @@ -20,7 +20,7 @@ if name == "nt": def find_git_on_windows(): - """find the path to the git executable on windows""" + """find the path to the git executable on Windows""" # first see if git is in the path try: check_output(["where", "/Q", "git"]) @@ -29,7 +29,7 @@ def find_git_on_windows(): # catch the exception thrown if git was not found except CalledProcessError: pass - # There are several locations git.exe may be hiding + # There are several locations where git.exe may be hiding possible_locations = [] # look in program files for msysgit if "PROGRAMFILES(X86)" in environ: @@ -38,7 +38,7 @@ def find_git_on_windows(): ) if "PROGRAMFILES" in environ: possible_locations.append("%s/Git/cmd/git.exe" % environ["PROGRAMFILES"]) - # look for the github version of git + # look for the GitHub version of git if "LOCALAPPDATA" in environ: github_dir = "%s/GitHub" % environ["LOCALAPPDATA"] if path.isdir(github_dir): diff --git a/metaflow/mflog/__init__.py b/metaflow/mflog/__init__.py index 691c1568ea8..c80b3a57c8f 100644 --- a/metaflow/mflog/__init__.py +++ b/metaflow/mflog/__init__.py @@ -4,6 +4,7 @@ from .mflog import refine, set_should_persist from metaflow.util import to_unicode +from metaflow.exception import MetaflowInternalError # Log source indicates the system that *minted the timestamp* # for the logline. This means that for a single task we can @@ -24,8 +25,8 @@ TASK_LOG_SOURCE = "task" # Loglines from all sources need to be merged together to -# produce a complete view of logs. Hence keep this list short -# since every items takes a DataStore access. +# produce a complete view of logs. Hence, keep this list short +# since each item takes a DataStore access. LOG_SOURCES = [RUNTIME_LOG_SOURCE, TASK_LOG_SOURCE] # BASH_MFLOG defines a bash function that outputs valid mflog @@ -43,18 +44,20 @@ BASH_SAVE_LOGS_ARGS = ["python", "-m", "metaflow.mflog.save_logs"] BASH_SAVE_LOGS = " ".join(BASH_SAVE_LOGS_ARGS) + # this function returns a bash expression that redirects stdout -# and stderr of the given command to mflog -def capture_output_to_mflog(command_and_args, var_transform=None): +# and stderr of the given bash expression to mflog.tee +def bash_capture_logs(bash_expr, var_transform=None): if var_transform is None: var_transform = lambda s: "$%s" % s - return "python -m metaflow.mflog.redirect_streams %s %s %s %s" % ( - TASK_LOG_SOURCE, - var_transform("MFLOG_STDOUT"), - var_transform("MFLOG_STDERR"), - command_and_args, + cmd = "python -m metaflow.mflog.tee %s %s" + parts = ( + bash_expr, + cmd % (TASK_LOG_SOURCE, var_transform("MFLOG_STDOUT")), + cmd % (TASK_LOG_SOURCE, var_transform("MFLOG_STDERR")), ) + return "(%s) 1>> >(%s) 2>> >(%s >&2)" % parts # update_delay determines how often logs should be uploaded to S3 @@ -76,8 +79,7 @@ def update_delay(secs_since_start): # this function is used to generate a Bash 'export' expression that -# sets environment variables that are used by 'redirect_streams' and -# 'save_logs'. +# sets environment variables that are used by 'tee' and 'save_logs'. # Note that we can't set the env vars statically, as some of them # may need to be evaluated during runtime def export_mflog_env_vars( @@ -142,3 +144,23 @@ def _available_logs(tail, stream, echo, should_persist=False): # tailed. _available_logs(stdout_tail, "stdout", echo) _available_logs(stderr_tail, "stderr", echo) + + +def get_log_tailer(log_url, datastore_type): + if datastore_type == "s3": + from metaflow.plugins.datatools.s3.s3tail import S3Tail + + return S3Tail(log_url) + elif datastore_type == "azure": + from metaflow.plugins.azure.azure_tail import AzureTail + + return AzureTail(log_url) + elif datastore_type == "gs": + from metaflow.plugins.gcp.gs_tail import GSTail + + return GSTail(log_url) + else: + raise MetaflowInternalError( + "Log tailing implementation missing for datastore type %s" + % (datastore_type,) + ) diff --git a/metaflow/mflog/mflog.py b/metaflow/mflog/mflog.py index fe29bdde4a2..fd04cb768ba 100644 --- a/metaflow/mflog/mflog.py +++ b/metaflow/mflog/mflog.py @@ -9,7 +9,7 @@ VERSION = b"0" -RE = b"(\[!)?" b"\[MFLOG\|" b"(0)\|" b"(.+?)Z\|" b"(.+?)\|" b"(.+?)\]" b"(.*)" +RE = rb"(\[!)?" rb"\[MFLOG\|" rb"(0)\|" rb"(.+?)Z\|" rb"(.+?)\|" rb"(.+?)\]" rb"(.*)" # the RE groups defined above must match the MFLogline fields below # except utc_timestamp, which is filled in by the parser based on utc_tstamp_str diff --git a/metaflow/mflog/redirect_streams.py b/metaflow/mflog/redirect_streams.py deleted file mode 100644 index 36aac03342c..00000000000 --- a/metaflow/mflog/redirect_streams.py +++ /dev/null @@ -1,54 +0,0 @@ -import os -import sys -import subprocess -import threading - -from .mflog import decorate - -# This script runs another process and captures stderr and stdout to a file, decorating -# lines with mflog metadata. -# -# Usage: redirect_streams SOURCE STDOUT_FILE STDERR_FILE PROGRAM ARG1 ARG2 ... - - -def reader_thread(SOURCE, dest_file, dest_stream, src): - with open(dest_file, mode="ab", buffering=0) as f: - if sys.version_info < (3, 0): - # Python 2 - for line in iter(sys.stdin.readline, ""): - # https://bugs.python.org/issue3907 - decorated = decorate(SOURCE, line) - f.write(decorated) - sys.stdout.write(line) - else: - # Python 3 - for line in src: - decorated = decorate(SOURCE, line) - f.write(decorated) - dest_stream.buffer.write(line) - - -if __name__ == "__main__": - SOURCE = sys.argv[1].encode("utf-8") - stdout_dest = sys.argv[2] - stderr_dest = sys.argv[3] - - p = subprocess.Popen( - sys.argv[4:], - env=os.environ, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - - stdout_reader = threading.Thread( - target=reader_thread, args=(SOURCE, stdout_dest, sys.stdout, p.stdout) - ) - stdout_reader.start() - stderr_reader = threading.Thread( - target=reader_thread, args=(SOURCE, stderr_dest, sys.stderr, p.stderr) - ) - stderr_reader.start() - rc = p.wait() - stdout_reader.join() - stderr_reader.join() - sys.exit(rc) diff --git a/metaflow/mflog/save_logs.py b/metaflow/mflog/save_logs.py index 4931766ae94..ea7ca673288 100644 --- a/metaflow/mflog/save_logs.py +++ b/metaflow/mflog/save_logs.py @@ -3,7 +3,8 @@ # This script is used to upload logs during task bootstrapping, so # it shouldn't have external dependencies besides Metaflow itself # (e.g. no click for parsing CLI args). -from metaflow.datastore import DATASTORES, FlowDataStore +from metaflow.datastore import FlowDataStore +from metaflow.plugins import DATASTORES from metaflow.util import Path from . import TASK_LOG_SOURCE @@ -23,7 +24,7 @@ def _read_file(path): paths = (os.environ["MFLOG_STDOUT"], os.environ["MFLOG_STDERR"]) flow_name, run_id, step_name, task_id = pathspec.split("/") - storage_impl = DATASTORES[ds_type] + storage_impl = [d for d in DATASTORES if d.TYPE == ds_type][0] if ds_root is None: def print_clean(line, **kwargs): diff --git a/metaflow/mflog/save_logs_periodically.py b/metaflow/mflog/save_logs_periodically.py index 32fc3d4c98e..d5618f95a30 100644 --- a/metaflow/mflog/save_logs_periodically.py +++ b/metaflow/mflog/save_logs_periodically.py @@ -4,8 +4,7 @@ import subprocess from threading import Thread -from metaflow.metaflow_profile import profile -from metaflow.sidecar import SidecarSubProcess +from metaflow.sidecar import MessageTypes from . import update_delay, BASH_SAVE_LOGS_ARGS @@ -16,10 +15,12 @@ def __init__(self): self._thread.start() def process_message(self, msg): - pass + if msg.msg_type == MessageTypes.SHUTDOWN: + self.is_alive = False - def shutdown(self): - self.is_alive = False + @classmethod + def get_worker(cls): + return cls def _update_loop(self): def _file_size(path): diff --git a/metaflow/monitor.py b/metaflow/monitor.py index ea33b5cd123..bd696333299 100644 --- a/metaflow/monitor.py +++ b/metaflow/monitor.py @@ -2,85 +2,67 @@ from contextlib import contextmanager -from .sidecar import SidecarSubProcess -from .sidecar_messages import Message, MessageTypes +from metaflow.sidecar import Message, MessageTypes, Sidecar COUNTER_TYPE = "COUNTER" GAUGE_TYPE = "GAUGE" -MEASURE_TYPE = "MEASURE" TIMER_TYPE = "TIMER" class NullMonitor(object): + TYPE = "nullSidecarMonitor" + def __init__(self, *args, **kwargs): - pass + # Currently passed flow and env as kwargs + self._sidecar = Sidecar(self.TYPE) def start(self): - pass - - @contextmanager - def count(self, name): - yield - - @contextmanager - def measure(self, name): - yield - - def gauge(self, gauge): - pass + return self._sidecar.start() def terminate(self): - pass + return self._sidecar.terminate() - -class Monitor(NullMonitor): - def __init__(self, monitor_type, env, flow_name): - # type: (str) -> None - self.sidecar_process = None - self.monitor_type = monitor_type - self.env_info = env.get_environment_info() - self.env_info["flow_name"] = flow_name - - def start(self): - if self.sidecar_process is None: - self.sidecar_process = SidecarSubProcess(self.monitor_type) + def send(self, msg): + # Arbitrary message sending. Useful if you want to override some different + # types of messages. + self._sidecar.send(msg) @contextmanager def count(self, name): - if self.sidecar_process is not None: - counter = Counter(name, self.env_info) + if self._sidecar.is_active: + counter = Counter(name) counter.increment() - payload = {"counter": counter.to_dict()} - msg = Message(MessageTypes.LOG_EVENT, payload) + payload = {"counter": counter.serialize()} + msg = Message(MessageTypes.BEST_EFFORT, payload) yield - self.sidecar_process.msg_handler(msg) + self._sidecar.send(msg) else: yield @contextmanager def measure(self, name): - if self.sidecar_process is not None: - timer = Timer(name + "_timer", self.env_info) - counter = Counter(name + "_counter", self.env_info) + if self._sidecar.is_active: + timer = Timer(name + "_timer") + counter = Counter(name + "_counter") timer.start() counter.increment() yield timer.end() - payload = {"counter": counter.to_dict(), "timer": timer.to_dict()} - msg = Message(MessageTypes.LOG_EVENT, payload) - self.sidecar_process.msg_handler(msg) + payload = {"counter": counter.serialize(), "timer": timer.serialize()} + msg = Message(MessageTypes.BEST_EFFORT, payload) + self._sidecar.send(msg) else: yield def gauge(self, gauge): - if self.sidecar_process is not None: - payload = {"gauge": gauge.to_dict()} - msg = Message(MessageTypes.LOG_EVENT, payload) - self.sidecar_process.msg_handler(msg) + if self._sidecar.is_active: + payload = {"gauge": gauge.serialize()} + msg = Message(MessageTypes.BEST_EFFORT, payload) + self._sidecar.send(msg) - def terminate(self): - if self.sidecar_process is not None: - self.sidecar_process.kill() + @classmethod + def get_worker(cls): + return None class Metric(object): @@ -88,84 +70,93 @@ class Metric(object): Abstract base class """ - def __init__(self, type, env): - self._env = env - self._type = type + def __init__(self, metric_type, name, context=None): + self._type = metric_type + self._name = name + self._context = context @property - def name(self): - raise NotImplementedError() + def metric_type(self): + return self._type @property - def flow_name(self): - return self._env["flow_name"] + def name(self): + return self._name @property - def env(self): - return self._env + def context(self): + return self._context + + @context.setter + def context(self, new_context): + self._context = new_context @property def value(self): raise NotImplementedError() - def set_env(self, env): - self._env = env - - def to_dict(self): - return { - "_env": self._env, - "_type": self._type, - } + def serialize(self): + # We purposefully do not serialize the context as it can be large; + # it will be transferred using a different mechanism and reset on the other + # end. + return {"_name": self._name, "_type": self._type} + + @classmethod + def deserialize(cls, value): + if value is None: + return None + metric_type = value.get("_type", "INVALID") + metric_name = value.get("_name", None) + metric_cls = _str_type_to_type.get(metric_type, None) + if metric_cls: + return metric_cls.deserialize(metric_name, value) + else: + raise NotImplementedError("Metric class %s is not supported" % metric_type) class Timer(Metric): - def __init__(self, name, env): - super(Timer, self).__init__(TIMER_TYPE, env) - self._name = name + def __init__(self, name, env=None): + super(Timer, self).__init__(TIMER_TYPE, name, env) self._start = 0 self._end = 0 - @property - def name(self): - return self._name - - def start(self): - self._start = time.time() - - def end(self): - self._end = time.time() - - def set_start(self, start): - self._start = start + def start(self, now=None): + if now is None: + now = time.time() + self._start = now - def set_end(self, end): - self._end = end + def end(self, now=None): + if now is None: + now = time.time() + self._end = now - def get_duration(self): + @property + def duration(self): return self._end - self._start @property def value(self): - return (self._end - self._start) * 1000 + return self.duration * 1000 + + def serialize(self): + parent_ser = super(Timer, self).serialize() + parent_ser["_start"] = self._start + parent_ser["_end"] = self._end + return parent_ser - def to_dict(self): - parent_dict = super(Timer, self).to_dict() - parent_dict["_name"] = self.name - parent_dict["_start"] = self._start - parent_dict["_end"] = self._end - return parent_dict + @classmethod + def deserialize(cls, metric_name, value): + t = Timer(metric_name) + t.start(value.get("_start", 0)) + t.end(value.get("_end", 0)) + return t class Counter(Metric): - def __init__(self, name, env): - super(Counter, self).__init__(COUNTER_TYPE, env) - self._name = name + def __init__(self, name, env=None): + super(Counter, self).__init__(COUNTER_TYPE, name, env) self._count = 0 - @property - def name(self): - return self._name - def increment(self): self._count += 1 @@ -176,23 +167,23 @@ def set_count(self, count): def value(self): return self._count - def to_dict(self): - parent_dict = super(Counter, self).to_dict() - parent_dict["_name"] = self.name - parent_dict["_count"] = self._count - return parent_dict + def serialize(self): + parent_ser = super(Counter, self).serialize() + parent_ser["_count"] = self._count + return parent_ser + + @classmethod + def deserialize(cls, metric_name, value): + c = Counter(metric_name) + c.set_count(value.get("_count", 0)) + return c class Gauge(Metric): - def __init__(self, name, env): - super(Gauge, self).__init__(GAUGE_TYPE, env) - self._name = name + def __init__(self, name, env=None): + super(Gauge, self).__init__(GAUGE_TYPE, name, env) self._value = 0 - @property - def name(self): - return self._name - def set_value(self, val): self._value = val @@ -203,47 +194,15 @@ def increment(self): def value(self): return self._value - def to_dict(self): - parent_dict = super(Gauge, self).to_dict() - parent_dict["_name"] = self.name - parent_dict["_value"] = self.value - return parent_dict - - -def deserialize_metric(metrics_dict): - if metrics_dict is None: - return - - type = metrics_dict.get("_type") - name = metrics_dict.get("_name") - if type == COUNTER_TYPE: - try: - counter = Counter(name, None) - counter.set_env(metrics_dict.get("_env")) - except Exception as ex: - return - - counter.set_count(metrics_dict.get("_count")) - return counter - elif type == TIMER_TYPE: - timer = Timer(name, None) - timer.set_start(metrics_dict.get("_start")) - timer.set_end(metrics_dict.get("_end")) - timer.set_env(metrics_dict.get("_env")) - return timer - elif type == GAUGE_TYPE: - gauge = Gauge(name, None) - gauge.set_env(metrics_dict.get("_env")) - gauge.set_value(metrics_dict.get("_value")) - return gauge - else: - raise NotImplementedError("UNSUPPORTED MESSAGE TYPE IN MONITOR") - - -def get_monitor_msg_type(msg): - if msg.payload.get("gauge") is not None: - return GAUGE_TYPE - if msg.payload.get("counter") is not None: - if msg.payload.get("timer") is not None: - return MEASURE_TYPE - return COUNTER_TYPE + def serialize(self): + parent_ser = super(Gauge, self).serialize() + parent_ser["_value"] = self._value + + @classmethod + def deserialize(cls, metric_name, value): + g = Gauge(metric_name) + g.set_value(value.get("_value", 0)) + return g + + +_str_type_to_type = {COUNTER_TYPE: Counter, GAUGE_TYPE: Gauge, TIMER_TYPE: Timer} diff --git a/metaflow/multicore_utils.py b/metaflow/multicore_utils.py index 486ea0c5a23..1ac81039c12 100644 --- a/metaflow/multicore_utils.py +++ b/metaflow/multicore_utils.py @@ -31,8 +31,8 @@ def _spawn(func, arg, dir): with NamedTemporaryFile(prefix="parallel_map_", dir=dir, delete=False) as tmpfile: output_file = tmpfile.name - # make sure stdout and stderr are flushed before forking. Otherwise - # we may print multiple copies of the same output + # Make sure stdout and stderr are flushed before forking, + # or else we may print multiple copies of the same output sys.stderr.flush() sys.stdout.flush() pid = os.fork() @@ -47,13 +47,13 @@ def _spawn(func, arg, dir): exit_code = 0 except: # we must not let any exceptions escape this function - # which might trigger unintended side-effects + # which might trigger unintended side effects traceback.print_exc() finally: sys.stderr.flush() sys.stdout.flush() # we can't use sys.exit(0) here since it raises SystemExit - # that may have unintended side-effects (e.g. triggering + # that may have unintended side effects (e.g. triggering # finally blocks). os._exit(exit_code) diff --git a/metaflow/package.py b/metaflow/package.py index cc1d1e0d85d..f68d15224ca 100644 --- a/metaflow/package.py +++ b/metaflow/package.py @@ -6,13 +6,14 @@ import json from io import BytesIO -from .extension_support import EXT_PKG -from .metaflow_config import DEFAULT_PACKAGE_SUFFIXES, METAFLOW_EXTENSIONS_ADDL_SUFFIXES +from .extension_support import EXT_PKG, package_mfext_all +from .metaflow_config import DEFAULT_PACKAGE_SUFFIXES from .exception import MetaflowException from .util import to_unicode from . import R DEFAULT_SUFFIXES_LIST = DEFAULT_PACKAGE_SUFFIXES.split(",") +METAFLOW_SUFFIXES_LIST = [".py", ".html", ".css", ".js"] class NonUniqueFileNameToFilePathMappingException(MetaflowException): @@ -21,7 +22,7 @@ class NonUniqueFileNameToFilePathMappingException(MetaflowException): def __init__(self, filename, file_paths, lineno=None): msg = ( "Filename %s included in the code package includes multiple different paths for the same name : %s.\n" - "The `filename` in the `add_to_package` decorator hook requires a unqiue `file_path` to `file_name` mapping" + "The `filename` in the `add_to_package` decorator hook requires a unique `file_path` to `file_name` mapping" % (filename, ", ".join(file_paths)) ) super().__init__(msg=msg, lineno=lineno) @@ -62,13 +63,6 @@ def __init__(self, flow, environment, echo, suffixes=DEFAULT_SUFFIXES_LIST): self.suffixes = list(set().union(suffixes, DEFAULT_SUFFIXES_LIST)) self.environment = environment self.metaflow_root = os.path.dirname(__file__) - try: - ext_package = importlib.import_module(EXT_PKG) - except ImportError as e: - self.metaflow_extensions_root = [] - else: - self.metaflow_extensions_root = list(ext_package.__path__) - self.metaflow_extensions_addl_suffixes = METAFLOW_EXTENSIONS_ADDL_SUFFIXES self.flow_name = flow.name self._flow = flow @@ -79,9 +73,9 @@ def __init__(self, flow, environment, echo, suffixes=DEFAULT_SUFFIXES_LIST): deco.package_init(flow, step.__name__, environment) self.blob = self._make() - def _walk(self, root, exclude_hidden=True, addl_suffixes=None): - if addl_suffixes is None: - addl_suffixes = [] + def _walk(self, root, exclude_hidden=True, suffixes=None): + if suffixes is None: + suffixes = [] root = to_unicode(root) # handle files/folder with non ascii chars prefixlen = len("%s/" % os.path.dirname(root)) for ( @@ -96,9 +90,7 @@ def _walk(self, root, exclude_hidden=True, addl_suffixes=None): for fname in files: if fname[0] == ".": continue - if any( - fname.endswith(suffix) for suffix in self.suffixes + addl_suffixes - ): + if any(fname.endswith(suffix) for suffix in suffixes): p = os.path.join(path, fname) yield p, p[prefixlen:] @@ -109,17 +101,16 @@ def path_tuples(self): """ # We want the following contents in the tarball # Metaflow package itself - for path_tuple in self._walk(self.metaflow_root, exclude_hidden=False): + for path_tuple in self._walk( + self.metaflow_root, exclude_hidden=False, suffixes=METAFLOW_SUFFIXES_LIST + ): + yield path_tuple + + # Metaflow extensions; for now, we package *all* extensions but this may change + # at a later date; it is possible to call `package_mfext_package` instead of + # `package_mfext_all` + for path_tuple in package_mfext_all(): yield path_tuple - # Metaflow customization if any - if self.metaflow_extensions_root: - for root in self.metaflow_extensions_root: - for path_tuple in self._walk( - root, - exclude_hidden=False, - addl_suffixes=self.metaflow_extensions_addl_suffixes, - ): - yield path_tuple # Any custom packages exposed via decorators deco_module_paths = {} @@ -142,7 +133,9 @@ def path_tuples(self): yield path_tuple if R.use_r(): # the R working directory - for path_tuple in self._walk("%s/" % R.working_dir()): + for path_tuple in self._walk( + "%s/" % R.working_dir(), suffixes=self.suffixes + ): yield path_tuple # the R package for path_tuple in R.package_paths(): @@ -150,7 +143,7 @@ def path_tuples(self): else: # the user's working directory flowdir = os.path.dirname(os.path.abspath(sys.argv[0])) + "/" - for path_tuple in self._walk(flowdir): + for path_tuple in self._walk(flowdir, suffixes=self.suffixes): yield path_tuple def _add_info(self, tar): diff --git a/metaflow/parameters.py b/metaflow/parameters.py index ac8173620aa..4f1a6ccb485 100644 --- a/metaflow/parameters.py +++ b/metaflow/parameters.py @@ -22,7 +22,13 @@ # breaking backwards compatibility but don't remove any fields! ParameterContext = namedtuple( "ParameterContext", - ["flow_name", "user_name", "parameter_name", "logger", "ds_type"], + [ + "flow_name", + "user_name", + "parameter_name", + "logger", + "ds_type", + ], ) # currently we execute only one flow per process, so we can treat @@ -55,9 +61,9 @@ def __repr__(self): class DeployTimeField(object): """ This a wrapper object for a user-defined function that is called - at the deploy time to populate fields in a Parameter. The wrapper + at deploy time to populate fields in a Parameter. The wrapper is needed to make Click show the actual value returned by the - function instead of a function pointer in its help text. Also this + function instead of a function pointer in its help text. Also, this object curries the context argument for the function, and pretty prints any exceptions that occur during evaluation. """ @@ -83,23 +89,37 @@ def __init__( if self.print_representation is None: self.print_representation = str(self.fun) - def __call__(self, full_evaluation=False): - # full_evaluation is True if there will be no further "convert" called - # by click and the parameter should be fully evaluated. + def __call__(self, deploy_time=False): + # This is called in two ways: + # - through the normal Click default parameter evaluation: if a default + # value is a callable, Click will call it without any argument. In other + # words, deploy_time=False. This happens for a normal "run" or the "trigger" + # functions for step-functions for example. Anything that has the + # @add_custom_parameters decorator will trigger this. Once click calls this, + # it will then pass the resulting value to the convert() functions for the + # type for that Parameter. + # - by deploy_time_eval which is invoked to process the parameters at + # deploy_time and outside of click processing (ie: at that point, Click + # is not involved since anytime deploy_time_eval is called, no custom parameters + # have been added). In that situation, deploy_time will be True. Note that in + # this scenario, the value should be something that can be converted to JSON. + # The deploy_time value can therefore be used to determine which type of + # processing is requested. ctx = context_proto._replace(parameter_name=self.parameter_name) try: try: - # Not all functions take two arguments - val = self.fun(ctx, full_evaluation) + # Most user-level functions may not care about the deploy_time parameter + # but IncludeFile does. + val = self.fun(ctx, deploy_time) except TypeError: val = self.fun(ctx) except: raise ParameterFieldFailed(self.parameter_name, self.field) else: - return self._check_type(val) + return self._check_type(val, deploy_time) - def _check_type(self, val): - # it is easy to introduce a deploy-time function that that accidentally + def _check_type(self, val, deploy_time): + # it is easy to introduce a deploy-time function that accidentally # returns a value whose type is not compatible with what is defined # in Parameter. Let's catch those mistakes early here, instead of # showing a cryptic stack trace later. @@ -120,6 +140,15 @@ def _check_type(self, val): raise ParameterFieldTypeMismatch(msg) return str(val) if self.return_str else val else: + if deploy_time: + try: + if not is_stringish(val): + val = json.dumps(val) + except TypeError: + msg += "Expected a JSON-encodable object or a string." + raise ParameterFieldTypeMismatch(msg) + return val + # If not deploy_time, we expect a string if not is_stringish(val): msg += "Expected a string." raise ParameterFieldTypeMismatch(msg) @@ -142,7 +171,7 @@ def __repr__(self): def deploy_time_eval(value): if isinstance(value, DeployTimeField): - return value(full_evaluation=True) + return value(deploy_time=True) else: return value @@ -159,7 +188,76 @@ def set_parameter_context(flow_name, echo, datastore): ) +class DelayedEvaluationParameter(object): + """ + This is a very simple wrapper to allow parameter "conversion" to be delayed until + the `_set_constants` function in FlowSpec. Typically, parameters are converted + by click when the command line option is processed. For some parameters, like + IncludeFile, this is too early as it would mean we would trigger the upload + of the file too early. If a parameter converts to a DelayedEvaluationParameter + object through the usual click mechanisms, `_set_constants` knows to invoke the + __call__ method on that DelayedEvaluationParameter; in that case, the __call__ + method is invoked without any parameter. The return_str parameter will be used + by schedulers when they need to convert DelayedEvaluationParameters to a + string to store them + """ + + def __init__(self, name, field, fun): + self._name = name + self._field = field + self._fun = fun + + def __call__(self, return_str=False): + try: + return self._fun(return_str=return_str) + except Exception as e: + raise ParameterFieldFailed(self._name, self._field) + + class Parameter(object): + """ + Defines a parameter for a flow. + + Parameters must be instantiated as class variables in flow classes, e.g. + ``` + class MyFlow(FlowSpec): + param = Parameter('myparam') + ``` + in this case, the parameter is specified on the command line as + ``` + python myflow.py run --myparam=5 + ``` + and its value is accessible through a read-only artifact like this: + ``` + print(self.param == 5) + ``` + Note that the user-visible parameter name, `myparam` above, can be + different from the artifact name, `param` above. + + The parameter value is converted to a Python type based on the `type` + argument or to match the type of `default`, if it is set. + + Parameters + ---------- + name : str + User-visible parameter name. + default : str or float or int or bool or `JSONType` or a function. + Default value for the parameter. Use a special `JSONType` class to + indicate that the value must be a valid JSON object. A function + implies that the parameter corresponds to a *deploy-time parameter*. + The type of the default value is used as the parameter `type`. + type : type + If `default` is not specified, define the parameter type. Specify + one of `str`, `float`, `int`, `bool`, or `JSONType` (default: str). + help : str + Help text to show in `run --help`. + required : bool + Require that the user specified a value for the parameter. + `required=True` implies that the `default` is not used. + show_default : bool + If True, show the default value in the help text (default: True). + """ + def __init__(self, name, **kwargs): self.name = name self.kwargs = kwargs diff --git a/metaflow/plugins/__init__.py b/metaflow/plugins/__init__.py index b3039c3721a..bb5c52ea746 100644 --- a/metaflow/plugins/__init__.py +++ b/metaflow/plugins/__init__.py @@ -1,4 +1,5 @@ import sys +import traceback import types @@ -14,7 +15,10 @@ def _merge_lists(base, overrides, attr): def _merge_funcs(base_func, override_func): - r = base_func() + override_func() + # IMPORTANT: This is a `get_plugin_cli` type of function, and we need to *delay* + # evaluation of it until after the flowspec is loaded. + old_default = base_func.__defaults__[0] + r = lambda: base_func(old_default) + override_func() base_func.__defaults__ = (r,) @@ -43,7 +47,7 @@ def _merge_funcs(base_func, override_func): [], lambda base, overrides: _merge_lists(base, overrides, "name"), ), - "get_plugin_cli": (lambda l=None: [] if l is None else l, _merge_funcs), + "get_plugin_cli": (lambda l=None: [] if l is None else l(), _merge_funcs), } @@ -63,6 +67,11 @@ def _merge_funcs(base_func, override_func): v[1](_ext_plugins[k], module_override) except Exception as e: _ext_debug("\tWARNING: ignoring all plugins due to error during import: %s" % e) + print( + "WARNING: Plugins did not load -- ignoring all of them which may not " + "be what you want: %s" % e + ) + traceback.print_exc() _ext_plugins = {k: v[0] for k, v in _expected_extensions.items()} _ext_debug("\tWill import the following plugins: %s" % str(_ext_plugins)) @@ -77,9 +86,12 @@ def get_plugin_cli(): # Add new CLI commands in this list from . import package_cli from .aws.batch import batch_cli - from .aws.eks import kubernetes_cli + from .kubernetes import kubernetes_cli from .aws.step_functions import step_functions_cli + from .airflow import airflow_cli + from .argo import argo_workflows_cli from .cards import card_cli + from . import tag_cli return _ext_plugins["get_plugin_cli"]() + [ package_cli.cli, @@ -87,6 +99,9 @@ def get_plugin_cli(): card_cli.cli, kubernetes_cli.cli, step_functions_cli.cli, + airflow_cli.cli, + argo_workflows_cli.cli, + tag_cli.cli, ] @@ -98,7 +113,8 @@ def get_plugin_cli(): from .retry_decorator import RetryDecorator from .resources_decorator import ResourcesDecorator from .aws.batch.batch_decorator import BatchDecorator -from .aws.eks.kubernetes_decorator import KubernetesDecorator +from .kubernetes.kubernetes_decorator import KubernetesDecorator +from .argo.argo_workflows_decorator import ArgoWorkflowsInternalDecorator from .aws.step_functions.step_functions_decorator import StepFunctionsInternalDecorator from .test_unbounded_foreach_decorator import ( InternalTestUnboundedForeachDecorator, @@ -107,6 +123,7 @@ def get_plugin_cli(): from .conda.conda_step_decorator import CondaStepDecorator from .cards.card_decorator import CardDecorator from .frameworks.pytorch import PytorchParallelDecorator +from .airflow.airflow_decorator import AirflowInternalDecorator STEP_DECORATORS = [ @@ -123,9 +140,19 @@ def get_plugin_cli(): ParallelDecorator, PytorchParallelDecorator, InternalTestUnboundedForeachDecorator, + AirflowInternalDecorator, + ArgoWorkflowsInternalDecorator, ] _merge_lists(STEP_DECORATORS, _ext_plugins["STEP_DECORATORS"], "name") +# Datastores +from .datastores.azure_storage import AzureStorage +from .datastores.gs_storage import GSStorage +from .datastores.local_storage import LocalStorage +from .datastores.s3_storage import S3Storage + +DATASTORES = [AzureStorage, GSStorage, LocalStorage, S3Storage] + # Add Conda environment from .conda.conda_environment import CondaEnvironment @@ -146,7 +173,12 @@ def get_plugin_cli(): from .aws.step_functions.schedule_decorator import ScheduleDecorator from .project_decorator import ProjectDecorator -FLOW_DECORATORS = [CondaFlowDecorator, ScheduleDecorator, ProjectDecorator] + +FLOW_DECORATORS = [ + CondaFlowDecorator, + ScheduleDecorator, + ProjectDecorator, +] _merge_lists(FLOW_DECORATORS, _ext_plugins["FLOW_DECORATORS"], "name") # Cards @@ -182,7 +214,8 @@ def get_plugin_cli(): TestNonEditableCard, BlankCard, DefaultCardJSON, -] + MF_EXTERNAL_CARDS +] +_merge_lists(CARDS, MF_EXTERNAL_CARDS, "type") # Sidecars from ..mflog.save_logs_periodically import SaveLogsPeriodicallySidecar from metaflow.metadata.heartbeat import MetadataHeartBeat @@ -195,14 +228,22 @@ def get_plugin_cli(): # Add logger from .debug_logger import DebugEventLogger +from metaflow.event_logger import NullEventLogger -LOGGING_SIDECARS = {"debugLogger": DebugEventLogger, "nullSidecarLogger": None} +LOGGING_SIDECARS = { + DebugEventLogger.TYPE: DebugEventLogger, + NullEventLogger.TYPE: NullEventLogger, +} LOGGING_SIDECARS.update(_ext_plugins["LOGGING_SIDECARS"]) # Add monitor from .debug_monitor import DebugMonitor +from metaflow.monitor import NullMonitor -MONITOR_SIDECARS = {"debugMonitor": DebugMonitor, "nullSidecarMonitor": None} +MONITOR_SIDECARS = { + DebugMonitor.TYPE: DebugMonitor, + NullMonitor.TYPE: NullMonitor, +} MONITOR_SIDECARS.update(_ext_plugins["MONITOR_SIDECARS"]) SIDECARS.update(LOGGING_SIDECARS) diff --git a/metaflow/plugins/airflow/__init__.py b/metaflow/plugins/airflow/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/metaflow/plugins/airflow/airflow.py b/metaflow/plugins/airflow/airflow.py new file mode 100644 index 00000000000..5480a79c59e --- /dev/null +++ b/metaflow/plugins/airflow/airflow.py @@ -0,0 +1,693 @@ +from io import BytesIO +import json +import os +import random +import string +import sys +from datetime import datetime, timedelta +from metaflow.includefile import FilePathClass + +import metaflow.util as util +from metaflow.decorators import flow_decorators +from metaflow.exception import MetaflowException +from metaflow.metaflow_config import ( + SERVICE_HEADERS, + SERVICE_INTERNAL_URL, + CARD_S3ROOT, + DATASTORE_SYSROOT_S3, + DATATOOLS_S3ROOT, + KUBERNETES_SERVICE_ACCOUNT, + KUBERNETES_SECRETS, + AIRFLOW_KUBERNETES_STARTUP_TIMEOUT_SECONDS, + AZURE_STORAGE_BLOB_SERVICE_ENDPOINT, + DATASTORE_SYSROOT_AZURE, + CARD_AZUREROOT, + AIRFLOW_KUBERNETES_CONN_ID, +) +from metaflow.parameters import DelayedEvaluationParameter, deploy_time_eval +from metaflow.plugins.kubernetes.kubernetes import Kubernetes + +# TODO: Move chevron to _vendor +from metaflow.plugins.cards.card_modules import chevron +from metaflow.plugins.timeout_decorator import get_run_time_limit_for_task +from metaflow.util import dict_to_cli_options, get_username, compress_list +from metaflow.parameters import JSONTypeClass + +from . import airflow_utils +from .exception import AirflowException +from .airflow_utils import ( + TASK_ID_XCOM_KEY, + AirflowTask, + Workflow, + AIRFLOW_MACROS, +) +from metaflow import current + +AIRFLOW_DEPLOY_TEMPLATE_FILE = os.path.join(os.path.dirname(__file__), "dag.py") + + +class Airflow(object): + + TOKEN_STORAGE_ROOT = "mf.airflow" + + def __init__( + self, + name, + graph, + flow, + code_package_sha, + code_package_url, + metadata, + flow_datastore, + environment, + event_logger, + monitor, + production_token, + tags=None, + namespace=None, + username=None, + max_workers=None, + worker_pool=None, + description=None, + file_path=None, + workflow_timeout=None, + is_paused_upon_creation=True, + ): + self.name = name + self.graph = graph + self.flow = flow + self.code_package_sha = code_package_sha + self.code_package_url = code_package_url + self.metadata = metadata + self.flow_datastore = flow_datastore + self.environment = environment + self.event_logger = event_logger + self.monitor = monitor + self.tags = tags + self.namespace = namespace # this is the username space + self.username = username + self.max_workers = max_workers + self.description = description + self._file_path = file_path + _, self.graph_structure = self.graph.output_steps() + self.worker_pool = worker_pool + self.is_paused_upon_creation = is_paused_upon_creation + self.workflow_timeout = workflow_timeout + self.schedule = self._get_schedule() + self.parameters = self._process_parameters() + self.production_token = production_token + self.contains_foreach = self._contains_foreach() + + @classmethod + def get_existing_deployment(cls, name, flow_datastore): + _backend = flow_datastore._storage_impl + token_paths = _backend.list_content([cls.get_token_path(name)]) + if len(token_paths) == 0: + return None + + with _backend.load_bytes([token_paths[0]]) as get_results: + for _, path, _ in get_results: + if path is not None: + with open(path, "r") as f: + data = json.loads(f.read()) + return (data["owner"], data["production_token"]) + + @classmethod + def get_token_path(cls, name): + return os.path.join(cls.TOKEN_STORAGE_ROOT, name) + + @classmethod + def save_deployment_token(cls, owner, token, flow_datastore): + _backend = flow_datastore._storage_impl + _backend.save_bytes( + [ + ( + cls.get_token_path(token), + BytesIO( + bytes( + json.dumps({"production_token": token, "owner": owner}), + "utf-8", + ) + ), + ) + ], + overwrite=False, + ) + + def _get_schedule(self): + # Using the cron presets provided here : + # https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html?highlight=schedule%20interval#cron-presets + schedule = self.flow._flow_decorators.get("schedule") + if not schedule: + return None + if schedule.attributes["cron"]: + return schedule.attributes["cron"] + elif schedule.attributes["weekly"]: + return "@weekly" + elif schedule.attributes["hourly"]: + return "@hourly" + elif schedule.attributes["daily"]: + return "@daily" + return None + + def _get_retries(self, node): + max_user_code_retries = 0 + max_error_retries = 0 + foreach_default_retry = 1 + # Different decorators may have different retrying strategies, so take + # the max of them. + for deco in node.decorators: + user_code_retries, error_retries = deco.step_task_retry_count() + max_user_code_retries = max(max_user_code_retries, user_code_retries) + max_error_retries = max(max_error_retries, error_retries) + parent_is_foreach = any( # The immediate parent is a foreach node. + self.graph[n].type == "foreach" for n in node.in_funcs + ) + + if parent_is_foreach: + max_user_code_retries + foreach_default_retry + return max_user_code_retries, max_user_code_retries + max_error_retries + + def _get_retry_delay(self, node): + retry_decos = [deco for deco in node.decorators if deco.name == "retry"] + if len(retry_decos) > 0: + retry_mins = retry_decos[0].attributes["minutes_between_retries"] + return timedelta(minutes=int(retry_mins)) + return None + + def _process_parameters(self): + airflow_params = [] + type_transform_dict = { + int.__name__: "integer", + str.__name__: "string", + bool.__name__: "string", + float.__name__: "number", + } + + for var, param in self.flow._get_parameters(): + # Airflow requires defaults set for parameters. + value = deploy_time_eval(param.kwargs.get("default")) + # Setting airflow related param args. + airflow_param = dict( + name=param.name, + ) + if value is not None: + airflow_param["default"] = value + if param.kwargs.get("help"): + airflow_param["description"] = param.kwargs.get("help") + + # Since we will always have a default value and `deploy_time_eval` resolved that to an actual value + # we can just use the `default` to infer the object's type. + # This avoids parsing/identifying types like `JSONType` or `FilePathClass` + # which are returned by calling `param.kwargs.get("type")` + param_type = type(airflow_param["default"]) + + # extract the name of the type and resolve the type-name + # compatible with Airflow. + param_type_name = getattr(param_type, "__name__", None) + if param_type_name in type_transform_dict: + airflow_param["type"] = type_transform_dict[param_type_name] + + if param_type_name == bool.__name__: + airflow_param["default"] = str(airflow_param["default"]) + + airflow_params.append(airflow_param) + + return airflow_params + + def _compress_input_path( + self, + steps, + ): + """ + This function is meant to compress the input paths, and it specifically doesn't use + `metaflow.util.compress_list` under the hood. The reason is that the `AIRFLOW_MACROS.RUN_ID` is a complicated + macro string that doesn't behave nicely with `metaflow.util.decompress_list`, since the `decompress_util` + function expects a string which doesn't contain any delimiter characters and the run-id string does. Hence, we + have a custom compression string created via `_compress_input_path` function instead of `compress_list`. + """ + return "%s:" % (AIRFLOW_MACROS.RUN_ID) + ",".join( + self._make_input_path(step, only_task_id=True) for step in steps + ) + + def _make_foreach_input_path(self, step_name): + return ( + "%s/%s/:{{ task_instance.xcom_pull(task_ids='%s',key='%s') | join_list }}" + % ( + AIRFLOW_MACROS.RUN_ID, + step_name, + step_name, + TASK_ID_XCOM_KEY, + ) + ) + + def _make_input_path(self, step_name, only_task_id=False): + """ + This is set using the `airflow_internal` decorator to help pass state. + This will pull the `TASK_ID_XCOM_KEY` xcom which holds task-ids. + The key is set via the `MetaflowKubernetesOperator`. + """ + task_id_string = "/%s/{{ task_instance.xcom_pull(task_ids='%s',key='%s') }}" % ( + step_name, + step_name, + TASK_ID_XCOM_KEY, + ) + + if only_task_id: + return task_id_string + + return "%s%s" % (AIRFLOW_MACROS.RUN_ID, task_id_string) + + def _to_job(self, node): + """ + This function will transform the node's specification into Airflow compatible operator arguments. + Since this function is long, below is the summary of the two major duties it performs: + 1. Based on the type of the graph node (start/linear/foreach/join etc.) + it will decide how to set the input paths + 2. Based on node's decorator specification convert the information into + a job spec for the KubernetesPodOperator. + """ + # Add env vars from the optional @environment decorator. + env_deco = [deco for deco in node.decorators if deco.name == "environment"] + env = {} + if env_deco: + env = env_deco[0].attributes["vars"] + + # The below if/else block handles "input paths". + # Input Paths help manage dataflow across the graph. + if node.name == "start": + # POSSIBLE_FUTURE_IMPROVEMENT: + # We can extract metadata about the possible upstream sensor triggers. + # There is a previous commit (7bdf6) in the `airflow` branch that has `SensorMetaExtractor` class and + # associated MACRO we have built to handle this case if a metadata regarding the sensor is needed. + # Initialize parameters for the flow in the `start` step. + # `start` step has no upstream input dependencies aside from + # parameters. + + if len(self.parameters): + env["METAFLOW_PARAMETERS"] = AIRFLOW_MACROS.PARAMETERS + input_paths = None + else: + # If it is not the start node then we check if there are many paths + # converging into it or a single path. Based on that we set the INPUT_PATHS + if node.parallel_foreach: + raise AirflowException( + "Parallel steps are not supported yet with Airflow." + ) + is_foreach_join = ( + node.type == "join" + and self.graph[node.split_parents[-1]].type == "foreach" + ) + if is_foreach_join: + input_paths = self._make_foreach_input_path(node.in_funcs[0]) + + elif len(node.in_funcs) == 1: + # set input paths where this is only one parent node + # The parent-task-id is passed via the xcom; There is no other way to get that. + # One key thing about xcoms is that they are immutable and only accepted if the task + # doesn't fail. + # From airflow docs : + # "Note: If the first task run is not succeeded then on every retry task + # XComs will be cleared to make the task run idempotent." + input_paths = self._make_input_path(node.in_funcs[0]) + else: + # this is a split scenario where there can be more than one input paths. + input_paths = self._compress_input_path(node.in_funcs) + + # env["METAFLOW_INPUT_PATHS"] = input_paths + + env["METAFLOW_CODE_URL"] = self.code_package_url + env["METAFLOW_FLOW_NAME"] = self.flow.name + env["METAFLOW_STEP_NAME"] = node.name + env["METAFLOW_OWNER"] = self.username + + metadata_env = self.metadata.get_runtime_environment("airflow") + env.update(metadata_env) + + metaflow_version = self.environment.get_environment_info() + metaflow_version["flow_name"] = self.graph.name + metaflow_version["production_token"] = self.production_token + env["METAFLOW_VERSION"] = json.dumps(metaflow_version) + + # Extract the k8s decorators for constructing the arguments of the K8s Pod Operator on Airflow. + k8s_deco = [deco for deco in node.decorators if deco.name == "kubernetes"][0] + user_code_retries, _ = self._get_retries(node) + retry_delay = self._get_retry_delay(node) + # This sets timeouts for @timeout decorators. + # The timeout is set as "execution_timeout" for an airflow task. + runtime_limit = get_run_time_limit_for_task(node.decorators) + + k8s = Kubernetes(self.flow_datastore, self.metadata, self.environment) + user = util.get_username() + + labels = { + "app": "metaflow", + "app.kubernetes.io/name": "metaflow-task", + "app.kubernetes.io/part-of": "metaflow", + "app.kubernetes.io/created-by": user, + # Question to (savin) : Should we have username set over here for created by since it is the + # airflow installation that is creating the jobs. + # Technically the "user" is the stakeholder but should these labels be present. + } + additional_mf_variables = { + "METAFLOW_CODE_SHA": self.code_package_sha, + "METAFLOW_CODE_URL": self.code_package_url, + "METAFLOW_CODE_DS": self.flow_datastore.TYPE, + "METAFLOW_USER": user, + "METAFLOW_SERVICE_URL": SERVICE_INTERNAL_URL, + "METAFLOW_SERVICE_HEADERS": json.dumps(SERVICE_HEADERS), + "METAFLOW_DATASTORE_SYSROOT_S3": DATASTORE_SYSROOT_S3, + "METAFLOW_DATATOOLS_S3ROOT": DATATOOLS_S3ROOT, + "METAFLOW_DEFAULT_DATASTORE": "s3", + "METAFLOW_DEFAULT_METADATA": "service", + "METAFLOW_KUBERNETES_WORKLOAD": str( + 1 + ), # This is used by kubernetes decorator. + "METAFLOW_RUNTIME_ENVIRONMENT": "kubernetes", + "METAFLOW_CARD_S3ROOT": CARD_S3ROOT, + "METAFLOW_RUN_ID": AIRFLOW_MACROS.RUN_ID, + "METAFLOW_AIRFLOW_TASK_ID": AIRFLOW_MACROS.create_task_id( + self.contains_foreach + ), + "METAFLOW_AIRFLOW_DAG_RUN_ID": AIRFLOW_MACROS.AIRFLOW_RUN_ID, + "METAFLOW_AIRFLOW_JOB_ID": AIRFLOW_MACROS.AIRFLOW_JOB_ID, + "METAFLOW_PRODUCTION_TOKEN": self.production_token, + "METAFLOW_ATTEMPT_NUMBER": AIRFLOW_MACROS.ATTEMPT, + } + env[ + "METAFLOW_AZURE_STORAGE_BLOB_SERVICE_ENDPOINT" + ] = AZURE_STORAGE_BLOB_SERVICE_ENDPOINT + env["METAFLOW_DATASTORE_SYSROOT_AZURE"] = DATASTORE_SYSROOT_AZURE + env["METAFLOW_CARD_AZUREROOT"] = CARD_AZUREROOT + env.update(additional_mf_variables) + + service_account = ( + KUBERNETES_SERVICE_ACCOUNT + if k8s_deco.attributes["service_account"] is None + else k8s_deco.attributes["service_account"] + ) + k8s_namespace = ( + k8s_deco.attributes["namespace"] + if k8s_deco.attributes["namespace"] is not None + else "default" + ) + + resources = dict( + requests={ + "cpu": k8s_deco.attributes["cpu"], + "memory": "%sM" % str(k8s_deco.attributes["memory"]), + "ephemeral-storage": str(k8s_deco.attributes["disk"]), + } + ) + if k8s_deco.attributes["gpu"] is not None: + resources.update( + dict( + limits={ + "%s.com/gpu".lower() + % k8s_deco.attributes["gpu_vendor"]: str( + k8s_deco.attributes["gpu"] + ) + } + ) + ) + + annotations = { + "metaflow/production_token": self.production_token, + "metaflow/owner": self.username, + "metaflow/user": self.username, + "metaflow/flow_name": self.flow.name, + } + if current.get("project_name"): + annotations.update( + { + "metaflow/project_name": current.project_name, + "metaflow/branch_name": current.branch_name, + "metaflow/project_flow_name": current.project_flow_name, + } + ) + + k8s_operator_args = dict( + # like argo workflows we use step_name as name of container + name=node.name, + namespace=k8s_namespace, + service_account_name=service_account, + node_selector=k8s_deco.attributes["node_selector"], + cmds=k8s._command( + self.flow.name, + AIRFLOW_MACROS.RUN_ID, + node.name, + AIRFLOW_MACROS.create_task_id(self.contains_foreach), + AIRFLOW_MACROS.ATTEMPT, + code_package_url=self.code_package_url, + step_cmds=self._step_cli( + node, input_paths, self.code_package_url, user_code_retries + ), + ), + annotations=annotations, + image=k8s_deco.attributes["image"], + resources=resources, + execution_timeout=dict(seconds=runtime_limit), + retries=user_code_retries, + env_vars=[dict(name=k, value=v) for k, v in env.items() if v is not None], + labels=labels, + task_id=node.name, + startup_timeout_seconds=AIRFLOW_KUBERNETES_STARTUP_TIMEOUT_SECONDS, + get_logs=True, + do_xcom_push=True, + log_events_on_failure=True, + is_delete_operator_pod=True, + retry_exponential_backoff=False, # todo : should this be a arg we allow on CLI. not right now - there is an open ticket for this - maybe at some point we will. + reattach_on_restart=False, + secrets=[], + ) + if AIRFLOW_KUBERNETES_CONN_ID is not None: + k8s_operator_args["kubernetes_conn_id"] = AIRFLOW_KUBERNETES_CONN_ID + else: + k8s_operator_args["in_cluster"] = True + + if k8s_deco.attributes["secrets"]: + if isinstance(k8s_deco.attributes["secrets"], str): + k8s_operator_args["secrets"] = k8s_deco.attributes["secrets"].split(",") + elif isinstance(k8s_deco.attributes["secrets"], list): + k8s_operator_args["secrets"] = k8s_deco.attributes["secrets"] + if len(KUBERNETES_SECRETS) > 0: + k8s_operator_args["secrets"] += KUBERNETES_SECRETS.split(",") + + if retry_delay: + k8s_operator_args["retry_delay"] = dict(seconds=retry_delay.total_seconds()) + + return k8s_operator_args + + def _step_cli(self, node, paths, code_package_url, user_code_retries): + cmds = [] + + script_name = os.path.basename(sys.argv[0]) + executable = self.environment.executable(node.name) + + entrypoint = [executable, script_name] + + top_opts_dict = { + "with": [ + decorator.make_decorator_spec() + for decorator in node.decorators + if not decorator.statically_defined + ] + } + # FlowDecorators can define their own top-level options. They are + # responsible for adding their own top-level options and values through + # the get_top_level_options() hook. See similar logic in runtime.py. + for deco in flow_decorators(): + top_opts_dict.update(deco.get_top_level_options()) + + top_opts = list(dict_to_cli_options(top_opts_dict)) + + top_level = top_opts + [ + "--quiet", + "--metadata=%s" % self.metadata.TYPE, + "--environment=%s" % self.environment.TYPE, + "--datastore=%s" % self.flow_datastore.TYPE, + "--datastore-root=%s" % self.flow_datastore.datastore_root, + "--event-logger=%s" % self.event_logger.TYPE, + "--monitor=%s" % self.monitor.TYPE, + "--no-pylint", + "--with=airflow_internal", + ] + + if node.name == "start": + # We need a separate unique ID for the special _parameters task + task_id_params = "%s-params" % AIRFLOW_MACROS.create_task_id( + self.contains_foreach + ) + # Export user-defined parameters into runtime environment + param_file = "".join( + random.choice(string.ascii_lowercase) for _ in range(10) + ) + # Setup Parameters as environment variables which are stored in a dictionary. + export_params = ( + "python -m " + "metaflow.plugins.airflow.plumbing.set_parameters %s " + "&& . `pwd`/%s" % (param_file, param_file) + ) + # Setting parameters over here. + params = ( + entrypoint + + top_level + + [ + "init", + "--run-id %s" % AIRFLOW_MACROS.RUN_ID, + "--task-id %s" % task_id_params, + ] + ) + + # Assign tags to run objects. + if self.tags: + params.extend("--tag %s" % tag for tag in self.tags) + + # If the start step gets retried, we must be careful not to + # regenerate multiple parameters tasks. Hence, we check first if + # _parameters exists already. + exists = entrypoint + [ + # Dump the parameters task + "dump", + "--max-value-size=0", + "%s/_parameters/%s" % (AIRFLOW_MACROS.RUN_ID, task_id_params), + ] + cmd = "if ! %s >/dev/null 2>/dev/null; then %s && %s; fi" % ( + " ".join(exists), + export_params, + " ".join(params), + ) + cmds.append(cmd) + # set input paths for parameters + paths = "%s/_parameters/%s" % (AIRFLOW_MACROS.RUN_ID, task_id_params) + + step = [ + "step", + node.name, + "--run-id %s" % AIRFLOW_MACROS.RUN_ID, + "--task-id %s" % AIRFLOW_MACROS.create_task_id(self.contains_foreach), + "--retry-count %s" % AIRFLOW_MACROS.ATTEMPT, + "--max-user-code-retries %d" % user_code_retries, + "--input-paths %s" % paths, + ] + if self.tags: + step.extend("--tag %s" % tag for tag in self.tags) + if self.namespace is not None: + step.append("--namespace=%s" % self.namespace) + + parent_is_foreach = any( # The immediate parent is a foreach node. + self.graph[n].type == "foreach" for n in node.in_funcs + ) + if parent_is_foreach: + step.append("--split-index %s" % AIRFLOW_MACROS.FOREACH_SPLIT_INDEX) + + cmds.append(" ".join(entrypoint + top_level + step)) + return cmds + + def _contains_foreach(self): + for node in self.graph: + if node.type == "foreach": + return True + return False + + def compile(self): + # Visit every node of the flow and recursively build the state machine. + def _visit(node, workflow, exit_node=None): + parent_is_foreach = any( # Any immediate parent is a foreach node. + self.graph[n].type == "foreach" for n in node.in_funcs + ) + state = AirflowTask( + node.name, is_mapper_node=parent_is_foreach + ).set_operator_args(**self._to_job(node)) + if node.type == "end": + workflow.add_state(state) + + # Continue linear assignment within the (sub)workflow if the node + # doesn't branch or fork. + elif node.type in ("start", "linear", "join", "foreach"): + workflow.add_state(state) + _visit( + self.graph[node.out_funcs[0]], + workflow, + ) + + elif node.type == "split": + workflow.add_state(state) + for func in node.out_funcs: + _visit( + self.graph[func], + workflow, + ) + else: + raise AirflowException( + "Node type *%s* for step *%s* " + "is not currently supported by " + "Airflow." % (node.type, node.name) + ) + + return workflow + + # set max active tasks here , For more info check here : + # https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dag/index.html#airflow.models.dag.DAG + airflow_dag_args = ( + {} if self.max_workers is None else dict(max_active_tasks=self.max_workers) + ) + airflow_dag_args["is_paused_upon_creation"] = self.is_paused_upon_creation + + # workflow timeout should only be enforced if a dag is scheduled. + if self.workflow_timeout is not None and self.schedule is not None: + airflow_dag_args["dagrun_timeout"] = dict(seconds=self.workflow_timeout) + + workflow = Workflow( + dag_id=self.name, + default_args=self._create_defaults(), + description=self.description, + schedule_interval=self.schedule, + # `start_date` is a mandatory argument even though the documentation lists it as optional value + # Based on the code, Airflow will throw a `AirflowException` when `start_date` is not provided + # to a DAG : https://github.com/apache/airflow/blob/0527a0b6ce506434a23bc2a6f5ddb11f492fc614/airflow/models/dag.py#L2170 + start_date=datetime.now(), + tags=self.tags, + file_path=self._file_path, + graph_structure=self.graph_structure, + metadata=dict( + contains_foreach=self.contains_foreach, flow_name=self.flow.name + ), + **airflow_dag_args + ) + workflow = _visit(self.graph["start"], workflow) + + workflow.set_parameters(self.parameters) + return self._to_airflow_dag_file(workflow.to_dict()) + + def _to_airflow_dag_file(self, json_dag): + util_file = None + with open(airflow_utils.__file__) as f: + util_file = f.read() + with open(AIRFLOW_DEPLOY_TEMPLATE_FILE) as f: + return chevron.render( + f.read(), + dict( + # Converting the configuration to base64 so that there can be no indentation related issues that can be caused because of + # malformed strings / json. + config=json_dag, + utils=util_file, + deployed_on=str(datetime.now()), + ), + ) + + def _create_defaults(self): + defu_ = { + "owner": get_username(), + # If set on a task and the previous run of the task has failed, + # it will not run the task in the current DAG run. + "depends_on_past": False, + # TODO: Enable emails + "execution_timeout": timedelta(days=5), + "retry_delay": timedelta(seconds=200), + # check https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html?highlight=retry_delay#airflow.models.baseoperator.BaseOperatorMeta + } + if self.worker_pool is not None: + defu_["pool"] = self.worker_pool + + return defu_ diff --git a/metaflow/plugins/airflow/airflow_cli.py b/metaflow/plugins/airflow/airflow_cli.py new file mode 100644 index 00000000000..5ac676978c2 --- /dev/null +++ b/metaflow/plugins/airflow/airflow_cli.py @@ -0,0 +1,434 @@ +import os +import re +import sys +import base64 +from metaflow import current, decorators +from metaflow._vendor import click +from metaflow.exception import MetaflowException, MetaflowInternalError +from metaflow.package import MetaflowPackage +from hashlib import sha1 +from metaflow.plugins.kubernetes.kubernetes_decorator import KubernetesDecorator +from metaflow.util import get_username, to_bytes, to_unicode + +from .airflow import Airflow +from .exception import AirflowException, NotSupportedException + +from metaflow.plugins.aws.step_functions.production_token import ( + load_token, + new_token, + store_token, +) + + +class IncorrectProductionToken(MetaflowException): + headline = "Incorrect production token" + + +VALID_NAME = re.compile("[^a-zA-Z0-9_\-\.]") + + +def resolve_token( + name, token_prefix, obj, authorize, given_token, generate_new_token, is_project +): + # 1) retrieve the previous deployment, if one exists + + workflow = Airflow.get_existing_deployment(name, obj.flow_datastore) + if workflow is None: + obj.echo( + "It seems this is the first time you are deploying *%s* to " + "Airflow." % name + ) + prev_token = None + else: + prev_user, prev_token = workflow + + # 2) authorize this deployment + if prev_token is not None: + if authorize is None: + authorize = load_token(token_prefix) + elif authorize.startswith("production:"): + authorize = authorize[11:] + + # we allow the user who deployed the previous version to re-deploy, + # even if they don't have the token + if prev_user != get_username() and authorize != prev_token: + obj.echo( + "There is an existing version of *%s* on Airflow which was " + "deployed by the user *%s*." % (name, prev_user) + ) + obj.echo( + "To deploy a new version of this flow, you need to use the same " + "production token that they used. " + ) + obj.echo( + "Please reach out to them to get the token. Once you have it, call " + "this command:" + ) + obj.echo(" airflow create --authorize MY_TOKEN", fg="green") + obj.echo( + 'See "Organizing Results" at docs.metaflow.org for more information ' + "about production tokens." + ) + raise IncorrectProductionToken( + "Try again with the correct production token." + ) + + # 3) do we need a new token or should we use the existing token? + if given_token: + if is_project: + # we rely on a known prefix for @project tokens, so we can't + # allow the user to specify a custom token with an arbitrary prefix + raise MetaflowException( + "--new-token is not supported for @projects. Use --generate-new-token " + "to create a new token." + ) + if given_token.startswith("production:"): + given_token = given_token[11:] + token = given_token + obj.echo("") + obj.echo("Using the given token, *%s*." % token) + elif prev_token is None or generate_new_token: + token = new_token(token_prefix, prev_token) + if token is None: + if prev_token is None: + raise MetaflowInternalError( + "We could not generate a new token. This is unexpected. " + ) + else: + raise MetaflowException( + "--generate-new-token option is not supported after using " + "--new-token. Use --new-token to make a new namespace." + ) + obj.echo("") + obj.echo("A new production token generated.") + Airflow.save_deployment_token(get_username(), token, obj.flow_datastore) + else: + token = prev_token + + obj.echo("") + obj.echo("The namespace of this production flow is") + obj.echo(" production:%s" % token, fg="green") + obj.echo( + "To analyze results of this production flow add this line in your notebooks:" + ) + obj.echo(' namespace("production:%s")' % token, fg="green") + obj.echo( + "If you want to authorize other people to deploy new versions of this flow to " + "Airflow, they need to call" + ) + obj.echo(" airflow create --authorize %s" % token, fg="green") + obj.echo("when deploying this flow to Airflow for the first time.") + obj.echo( + 'See "Organizing Results" at https://docs.metaflow.org/ for more ' + "information about production tokens." + ) + obj.echo("") + store_token(token_prefix, token) + + return token + + +@click.group() +def cli(): + pass + + +@cli.group(help="Commands related to Airflow.") +@click.option( + "--name", + default=None, + type=str, + help="Airflow DAG name. The flow name is used instead if this option is not " + "specified", +) +@click.pass_obj +def airflow(obj, name=None): + obj.check(obj.graph, obj.flow, obj.environment, pylint=obj.pylint) + obj.dag_name, obj.token_prefix, obj.is_project = resolve_dag_name(name) + + +@airflow.command(help="Compile a new version of this flow to Airflow DAG.") +@click.argument("file", required=True) +@click.option( + "--authorize", + default=None, + help="Authorize using this production token. You need this " + "when you are re-deploying an existing flow for the first " + "time. The token is cached in METAFLOW_HOME, so you only " + "need to specify this once.", +) +@click.option( + "--generate-new-token", + is_flag=True, + help="Generate a new production token for this flow. " + "This will move the production flow to a new namespace.", +) +@click.option( + "--new-token", + "given_token", + default=None, + help="Use the given production token for this flow. " + "This will move the production flow to the given namespace.", +) +@click.option( + "--tag", + "tags", + multiple=True, + default=None, + help="Annotate all objects produced by Airflow DAG executions " + "with the given tag. You can specify this option multiple " + "times to attach multiple tags.", +) +@click.option( + "--is-paused-upon-creation", + default=False, + is_flag=True, + help="Generated Airflow DAG is paused/unpaused upon creation.", +) +@click.option( + "--namespace", + "user_namespace", + default=None, + # TODO (savin): Identify the default namespace? + help="Change the namespace from the default to the given tag. " + "See run --help for more information.", +) +@click.option( + "--max-workers", + default=100, + show_default=True, + help="Maximum number of parallel processes.", +) +@click.option( + "--workflow-timeout", + default=None, + type=int, + help="Workflow timeout in seconds. Enforced only for scheduled DAGs.", +) +@click.option( + "--worker-pool", + default=None, + show_default=True, + help="Worker pool for Airflow DAG execution.", +) +@click.pass_obj +def create( + obj, + file, + authorize=None, + generate_new_token=False, + given_token=None, + tags=None, + is_paused_upon_creation=False, + user_namespace=None, + max_workers=None, + workflow_timeout=None, + worker_pool=None, +): + if os.path.abspath(sys.argv[0]) == os.path.abspath(file): + raise MetaflowException( + "Airflow DAG file name cannot be the same as flow file name" + ) + + # Validate if the workflow is correctly parsed. + _validate_workflow( + obj.flow, obj.graph, obj.flow_datastore, obj.metadata, workflow_timeout + ) + + obj.echo("Compiling *%s* to Airflow DAG..." % obj.dag_name, bold=True) + token = resolve_token( + obj.dag_name, + obj.token_prefix, + obj, + authorize, + given_token, + generate_new_token, + obj.is_project, + ) + + flow = make_flow( + obj, + obj.dag_name, + token, + tags, + is_paused_upon_creation, + user_namespace, + max_workers, + workflow_timeout, + worker_pool, + file, + ) + with open(file, "w") as f: + f.write(flow.compile()) + + obj.echo( + "DAG *{dag_name}* " + "for flow *{name}* compiled to " + "Airflow successfully.\n".format(dag_name=obj.dag_name, name=current.flow_name), + bold=True, + ) + + +def make_flow( + obj, + dag_name, + production_token, + tags, + is_paused_upon_creation, + namespace, + max_workers, + workflow_timeout, + worker_pool, + file, +): + # Attach @kubernetes. + decorators._attach_decorators(obj.flow, [KubernetesDecorator.name]) + + decorators._init_step_decorators( + obj.flow, obj.graph, obj.environment, obj.flow_datastore, obj.logger + ) + + # Save the code package in the flow datastore so that both user code and + # metaflow package can be retrieved during workflow execution. + obj.package = MetaflowPackage( + obj.flow, obj.environment, obj.echo, obj.package_suffixes + ) + package_url, package_sha = obj.flow_datastore.save_data( + [obj.package.blob], len_hint=1 + )[0] + + return Airflow( + dag_name, + obj.graph, + obj.flow, + package_sha, + package_url, + obj.metadata, + obj.flow_datastore, + obj.environment, + obj.event_logger, + obj.monitor, + production_token, + tags=tags, + namespace=namespace, + username=get_username(), + max_workers=max_workers, + worker_pool=worker_pool, + workflow_timeout=workflow_timeout, + description=obj.flow.__doc__, + file_path=file, + is_paused_upon_creation=is_paused_upon_creation, + ) + + +def _validate_foreach_constraints(graph): + # Todo :Invoke this function when we integrate `foreach`s + def traverse_graph(node, state): + if node.type == "foreach" and node.is_inside_foreach: + raise NotSupportedException( + "Step *%s* is a foreach step called within a foreach step. " + "This type of graph is currently not supported with Airflow." + % node.name + ) + + if node.type == "foreach": + state["foreach_stack"] = [node.name] + + if node.type in ("start", "linear", "join", "foreach"): + if node.type == "linear" and node.is_inside_foreach: + state["foreach_stack"].append(node.name) + + if len(state["foreach_stack"]) > 2: + raise NotSupportedException( + "The foreach step *%s* created by step *%s* needs to have an immediate join step. " + "Step *%s* is invalid since it is a linear step with a foreach. " + "This type of graph is currently not supported with Airflow." + % ( + state["foreach_stack"][1], + state["foreach_stack"][0], + state["foreach_stack"][-1], + ) + ) + + traverse_graph(graph[node.out_funcs[0]], state) + + elif node.type == "split": + for func in node.out_funcs: + traverse_graph(graph[func], state) + + traverse_graph(graph["start"], {}) + + +def _validate_workflow(flow, graph, flow_datastore, metadata, workflow_timeout): + seen = set() + for var, param in flow._get_parameters(): + # Throw an exception if the parameter is specified twice. + norm = param.name.lower() + if norm in seen: + raise MetaflowException( + "Parameter *%s* is specified twice. " + "Note that parameter names are " + "case-insensitive." % param.name + ) + seen.add(norm) + if "default" not in param.kwargs: + raise MetaflowException( + "Parameter *%s* does not have a " + "default value. " + "A default value is required for parameters when deploying flows on Airflow." + ) + # check for other compute related decorators. + for node in graph: + if node.parallel_foreach: + raise AirflowException( + "Deploying flows with @parallel decorator(s) " + "to Airflow is not supported currently." + ) + + if node.type == "foreach": + raise NotSupportedException( + "Step *%s* is a foreach step and Foreach steps are not currently supported with Airflow." + % node.name + ) + if any([d.name == "batch" for d in node.decorators]): + raise NotSupportedException( + "Step *%s* is marked for execution on AWS Batch with Airflow which isn't currently supported." + % node.name + ) + + if flow_datastore.TYPE not in ("azure", "s3"): + raise AirflowException( + 'Datastore of type "s3" or "azure" required with `airflow create`' + ) + + +def resolve_dag_name(name): + project = current.get("project_name") + is_project = False + + if project: + is_project = True + if name: + raise MetaflowException( + "--name is not supported for @projects. " "Use --branch instead." + ) + dag_name = current.project_flow_name + if dag_name and VALID_NAME.search(dag_name): + raise MetaflowException( + "Name '%s' contains invalid characters. Please construct a name using regex %s" + % (dag_name, VALID_NAME.pattern) + ) + project_branch = to_bytes(".".join((project, current.branch_name))) + token_prefix = ( + "mfprj-%s" + % to_unicode(base64.b32encode(sha1(project_branch).digest()))[:16] + ) + else: + if name and VALID_NAME.search(name): + raise MetaflowException( + "Name '%s' contains invalid characters. Please construct a name using regex %s" + % (name, VALID_NAME.pattern) + ) + dag_name = name if name else current.flow_name + token_prefix = dag_name + return dag_name, token_prefix.lower(), is_project diff --git a/metaflow/plugins/airflow/airflow_decorator.py b/metaflow/plugins/airflow/airflow_decorator.py new file mode 100644 index 00000000000..11bdecaaa8b --- /dev/null +++ b/metaflow/plugins/airflow/airflow_decorator.py @@ -0,0 +1,66 @@ +import json +import os +from metaflow.decorators import StepDecorator +from metaflow.metadata import MetaDatum + +from .airflow_utils import ( + TASK_ID_XCOM_KEY, + FOREACH_CARDINALITY_XCOM_KEY, +) + +K8S_XCOM_DIR_PATH = "/airflow/xcom" + + +def safe_mkdir(dir): + try: + os.makedirs(dir) + except FileExistsError: + pass + + +def push_xcom_values(xcom_dict): + safe_mkdir(K8S_XCOM_DIR_PATH) + with open(os.path.join(K8S_XCOM_DIR_PATH, "return.json"), "w") as f: + json.dump(xcom_dict, f) + + +class AirflowInternalDecorator(StepDecorator): + name = "airflow_internal" + + def task_pre_step( + self, + step_name, + task_datastore, + metadata, + run_id, + task_id, + flow, + graph, + retry_count, + max_user_code_retries, + ubf_context, + inputs, + ): + meta = {} + meta["airflow-dag-run-id"] = os.environ["METAFLOW_AIRFLOW_DAG_RUN_ID"] + meta["airflow-job-id"] = os.environ["METAFLOW_AIRFLOW_JOB_ID"] + entries = [ + MetaDatum( + field=k, value=v, type=k, tags=["attempt_id:{0}".format(retry_count)] + ) + for k, v in meta.items() + ] + + # Register book-keeping metadata for debugging. + metadata.register_metadata(run_id, step_name, task_id, entries) + + def task_finished( + self, step_name, flow, graph, is_task_ok, retry_count, max_user_code_retries + ): + # This will pass the xcom when the task finishes. + xcom_values = { + TASK_ID_XCOM_KEY: os.environ["METAFLOW_AIRFLOW_TASK_ID"], + } + if graph[step_name].type == "foreach": + xcom_values[FOREACH_CARDINALITY_XCOM_KEY] = flow._foreach_num_splits + push_xcom_values(xcom_values) diff --git a/metaflow/plugins/airflow/airflow_utils.py b/metaflow/plugins/airflow/airflow_utils.py new file mode 100644 index 00000000000..26e544e8d61 --- /dev/null +++ b/metaflow/plugins/airflow/airflow_utils.py @@ -0,0 +1,672 @@ +import hashlib +import json +import sys +import platform +from collections import defaultdict +from datetime import datetime, timedelta + + +TASK_ID_XCOM_KEY = "metaflow_task_id" +FOREACH_CARDINALITY_XCOM_KEY = "metaflow_foreach_cardinality" +FOREACH_XCOM_KEY = "metaflow_foreach_indexes" +RUN_HASH_ID_LEN = 12 +TASK_ID_HASH_LEN = 8 +RUN_ID_PREFIX = "airflow" +AIRFLOW_FOREACH_SUPPORT_VERSION = "2.3.0" +AIRFLOW_MIN_SUPPORT_VERSION = "2.2.0" +KUBERNETES_PROVIDER_FOREACH_VERSION = "4.2.0" + + +class KubernetesProviderNotFound(Exception): + headline = "Kubernetes provider not found" + + +class ForeachIncompatibleException(Exception): + headline = "Airflow version is incompatible to support Metaflow `foreach`s." + + +class IncompatibleVersionException(Exception): + headline = "Metaflow is incompatible with current version of Airflow." + + def __init__(self, version_number) -> None: + msg = ( + "Airflow version %s is incompatible with Metaflow. Metaflow requires Airflow a minimum version %s" + % (version_number, AIRFLOW_MIN_SUPPORT_VERSION) + ) + super().__init__(msg) + + +class IncompatibleKubernetesProviderVersionException(Exception): + headline = ( + "Kubernetes Provider version is incompatible with Metaflow `foreach`s. " + "Install the provider via " + "`%s -m pip install apache-airflow-providers-cncf-kubernetes==%s`" + ) % (sys.executable, KUBERNETES_PROVIDER_FOREACH_VERSION) + + +def create_absolute_version_number(version): + abs_version = None + # For all digits + if all(v.isdigit() for v in version.split(".")): + abs_version = sum( + [ + (10 ** (3 - idx)) * i + for idx, i in enumerate([int(v) for v in version.split(".")]) + ] + ) + # For first two digits + elif all(v.isdigit() for v in version.split(".")[:2]): + abs_version = sum( + [ + (10 ** (3 - idx)) * i + for idx, i in enumerate([int(v) for v in version.split(".")[:2]]) + ] + ) + return abs_version + + +def _validate_dynamic_mapping_compatibility(): + from airflow.version import version + + af_ver = create_absolute_version_number(version) + if af_ver is None or af_ver < create_absolute_version_number( + AIRFLOW_FOREACH_SUPPORT_VERSION + ): + ForeachIncompatibleException( + "Please install airflow version %s to use Airflow's Dynamic task mapping functionality." + % AIRFLOW_FOREACH_SUPPORT_VERSION + ) + + +def get_kubernetes_provider_version(): + try: + from airflow.providers.cncf.kubernetes.get_provider_info import ( + get_provider_info, + ) + except ImportError as e: + raise KubernetesProviderNotFound( + "This DAG utilizes `KubernetesPodOperator`. " + "Install the Airflow Kubernetes provider using " + "`%s -m pip install apache-airflow-providers-cncf-kubernetes`" + % sys.executable + ) + return get_provider_info()["versions"][0] + + +def _validate_minimum_airflow_version(): + from airflow.version import version + + af_ver = create_absolute_version_number(version) + if af_ver is None or af_ver < create_absolute_version_number( + AIRFLOW_MIN_SUPPORT_VERSION + ): + raise IncompatibleVersionException(version) + + +def _check_foreach_compatible_kubernetes_provider(): + provider_version = get_kubernetes_provider_version() + ver = create_absolute_version_number(provider_version) + if ver is None or ver < create_absolute_version_number( + KUBERNETES_PROVIDER_FOREACH_VERSION + ): + raise IncompatibleKubernetesProviderVersionException() + + +def datetimeparse(isotimestamp): + ver = int(platform.python_version_tuple()[0]) * 10 + int( + platform.python_version_tuple()[1] + ) + if ver >= 37: + return datetime.fromisoformat(isotimestamp) + else: + return datetime.strptime(isotimestamp, "%Y-%m-%dT%H:%M:%S.%f") + + +def get_xcom_arg_class(): + try: + from airflow import XComArg + except ImportError: + return None + return XComArg + + +class AIRFLOW_MACROS: + # run_id_creator is added via the `user_defined_filters` + RUN_ID = "%s-{{ [run_id, dag_run.dag_id] | run_id_creator }}" % RUN_ID_PREFIX + PARAMETERS = "{{ params | json_dump }}" + + STEPNAME = "{{ ti.task_id }}" + + # AIRFLOW_MACROS.TASK_ID will work for linear/branched workflows. + # ti.task_id is the stepname in metaflow code. + # AIRFLOW_MACROS.TASK_ID uses a jinja filter called `task_id_creator` which helps + # concatenate the string using a `/`. Since run-id will keep changing and stepname will be + # the same task id will change. Since airflow doesn't encourage dynamic rewriting of dags + # we can rename steps in a foreach with indexes (eg. `stepname-$index`) to create those steps. + # Hence : `foreach`s will require some special form of plumbing. + # https://stackoverflow.com/questions/62962386/can-an-airflow-task-dynamically-generate-a-dag-at-runtime + TASK_ID = ( + "%s-{{ [run_id, ti.task_id, dag_run.dag_id] | task_id_creator }}" + % RUN_ID_PREFIX + ) + + FOREACH_TASK_ID = ( + "%s-{{ [run_id, ti.task_id, dag_run.dag_id, ti.map_index] | task_id_creator }}" + % RUN_ID_PREFIX + ) + + # Airflow run_ids are of the form : "manual__2022-03-15T01:26:41.186781+00:00" + # Such run-ids break the `metaflow.util.decompress_list`; this is why we hash the runid + # We do `echo -n` because it emits line breaks, and we don't want to consider that, since we want same hash value + # when retrieved in python. + RUN_ID_SHELL = ( + "%s-$(echo -n {{ run_id }}-{{ dag_run.dag_id }} | md5sum | awk '{print $1}' | awk '{print substr ($0, 0, %s)}')" + % (RUN_ID_PREFIX, str(RUN_HASH_ID_LEN)) + ) + + ATTEMPT = "{{ task_instance.try_number - 1 }}" + + AIRFLOW_RUN_ID = "{{ run_id }}" + + AIRFLOW_JOB_ID = "{{ ti.job_id }}" + + FOREACH_SPLIT_INDEX = "{{ ti.map_index }}" + + @classmethod + def create_task_id(cls, is_foreach): + if is_foreach: + return cls.FOREACH_TASK_ID + else: + return cls.TASK_ID + + @classmethod + def pathspec(cls, flowname, is_foreach=False): + return "%s/%s/%s/%s" % ( + flowname, + cls.RUN_ID, + cls.STEPNAME, + cls.create_task_id(is_foreach), + ) + + +def run_id_creator(val): + # join `[dag-id,run-id]` of airflow dag. + return hashlib.md5("-".join([str(x) for x in val]).encode("utf-8")).hexdigest()[ + :RUN_HASH_ID_LEN + ] + + +def task_id_creator(val): + # join `[dag-id,run-id]` of airflow dag. + return hashlib.md5("-".join([str(x) for x in val]).encode("utf-8")).hexdigest()[ + :TASK_ID_HASH_LEN + ] + + +def id_creator(val, hash_len): + # join `[dag-id,run-id]` of airflow dag. + return hashlib.md5("-".join([str(x) for x in val]).encode("utf-8")).hexdigest()[ + :hash_len + ] + + +def json_dump(val): + return json.dumps(val) + + +class AirflowDAGArgs(object): + + # `_arg_types` is a dictionary which represents the types of the arguments of an Airflow `DAG`. + # `_arg_types` is used when parsing types back from the configuration json. + # It doesn't cover all the arguments but covers many of the important one which can come from the cli. + _arg_types = { + "dag_id": str, + "description": str, + "schedule_interval": str, + "start_date": datetime, + "catchup": bool, + "tags": list, + "dagrun_timeout": timedelta, + "default_args": { + "owner": str, + "depends_on_past": bool, + "email": list, + "email_on_failure": bool, + "email_on_retry": bool, + "retries": int, + "retry_delay": timedelta, + "queue": str, # which queue to target when running this job. Not all executors implement queue management, the CeleryExecutor does support targeting specific queues. + "pool": str, # the slot pool this task should run in, slot pools are a way to limit concurrency for certain tasks + "priority_weight": int, + "wait_for_downstream": bool, + "sla": timedelta, + "execution_timeout": timedelta, + "trigger_rule": str, + }, + } + + # Reference for user_defined_filters : https://stackoverflow.com/a/70175317 + filters = dict( + task_id_creator=lambda v: task_id_creator(v), + json_dump=lambda val: json_dump(val), + run_id_creator=lambda val: run_id_creator(val), + join_list=lambda x: ",".join(list(x)), + ) + + def __init__(self, **kwargs): + self._args = kwargs + + @property + def arguments(self): + return dict(**self._args, user_defined_filters=self.filters) + + def serialize(self): + def parse_args(dd): + data_dict = {} + for k, v in dd.items(): + if isinstance(v, dict): + data_dict[k] = parse_args(v) + elif isinstance(v, datetime): + data_dict[k] = v.isoformat() + elif isinstance(v, timedelta): + data_dict[k] = dict(seconds=v.total_seconds()) + else: + data_dict[k] = v + return data_dict + + return parse_args(self._args) + + @classmethod + def deserialize(cls, data_dict): + def parse_args(dd, type_check_dict): + kwrgs = {} + for k, v in dd.items(): + if k not in type_check_dict: + kwrgs[k] = v + elif isinstance(v, dict) and isinstance(type_check_dict[k], dict): + kwrgs[k] = parse_args(v, type_check_dict[k]) + elif type_check_dict[k] == datetime: + kwrgs[k] = datetimeparse(v) + elif type_check_dict[k] == timedelta: + kwrgs[k] = timedelta(**v) + else: + kwrgs[k] = v + return kwrgs + + return cls(**parse_args(data_dict, cls._arg_types)) + + +def _kubernetes_pod_operator_args(operator_args): + from kubernetes import client + + from airflow.kubernetes.secret import Secret + + # Set dynamic env variables like run-id, task-id etc from here. + secrets = [ + Secret("env", secret, secret) for secret in operator_args.get("secrets", []) + ] + args = operator_args + args.update( + { + "secrets": secrets, + # Question for (savin): + # Default timeout in airflow is 120. I can remove `startup_timeout_seconds` for now. how should we expose it to the user? + } + ) + # We need to explicitly add the `client.V1EnvVar` over here because + # `pod_runtime_info_envs` doesn't accept arguments in dictionary form and strictly + # Requires objects of type `client.V1EnvVar` + additional_env_vars = [ + client.V1EnvVar( + name=k, + value_from=client.V1EnvVarSource( + field_ref=client.V1ObjectFieldSelector(field_path=str(v)) + ), + ) + for k, v in { + "METAFLOW_KUBERNETES_POD_NAMESPACE": "metadata.namespace", + "METAFLOW_KUBERNETES_POD_NAME": "metadata.name", + "METAFLOW_KUBERNETES_POD_ID": "metadata.uid", + "METAFLOW_KUBERNETES_SERVICE_ACCOUNT_NAME": "spec.serviceAccountName", + }.items() + ] + args["pod_runtime_info_envs"] = additional_env_vars + + resources = args.get("resources") + # KubernetesPodOperator version 4.2.0 renamed `resources` to + # `container_resources` (https://github.com/apache/airflow/pull/24673) / (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85) + # This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to + # deprecate the `resources` argument. Hence, the below code path checks for the version of `KubernetesPodOperator` + # and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. + # If it is > 4.2.0 then we set the argument as `container_resources` + # The `resources` argument of `KubernetesPodOperator` is going to be deprecated soon in the future. + # So we will only use it for `KubernetesPodOperator` version < 4.2.0 + # The `resources` argument will also not work for `foreach`s. + provider_version = get_kubernetes_provider_version() + k8s_op_ver = create_absolute_version_number(provider_version) + if k8s_op_ver is None or k8s_op_ver < create_absolute_version_number( + KUBERNETES_PROVIDER_FOREACH_VERSION + ): + # Since the provider version is less than `4.2.0` so we need to use the `resources` argument + # We need to explicitly parse `resources`/`container_resources` to `k8s.V1ResourceRequirements`, + # otherwise airflow tries to parse dictionaries to `airflow.providers.cncf.kubernetes.backcompat.pod.Resources` + # object via `airflow.providers.cncf.kubernetes.backcompat.backward_compat_converts.convert_resources` function. + # This fails many times since the dictionary structure it expects is not the same as + # `client.V1ResourceRequirements`. + args["resources"] = client.V1ResourceRequirements( + requests=resources["requests"], + limits=None if "limits" not in resources else resources["limits"], + ) + else: # since the provider version is greater than `4.2.0` so should use the `container_resources` argument + args["container_resources"] = client.V1ResourceRequirements( + requests=resources["requests"], + limits=None if "limits" not in resources else resources["limits"], + ) + del args["resources"] + + if operator_args.get("execution_timeout"): + args["execution_timeout"] = timedelta( + **operator_args.get( + "execution_timeout", + ) + ) + if operator_args.get("retry_delay"): + args["retry_delay"] = timedelta(**operator_args.get("retry_delay")) + return args + + +def get_metaflow_kubernetes_operator(): + try: + from airflow.contrib.operators.kubernetes_pod_operator import ( + KubernetesPodOperator, + ) + except ImportError: + try: + from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import ( + KubernetesPodOperator, + ) + except ImportError as e: + raise KubernetesProviderNotFound( + "This DAG utilizes `KubernetesPodOperator`. " + "Install the Airflow Kubernetes provider using " + "`%s -m pip install apache-airflow-providers-cncf-kubernetes`" + % sys.executable + ) + + class MetaflowKubernetesOperator(KubernetesPodOperator): + """ + ## Why Inherit the `KubernetesPodOperator` class ? + + Two key reasons : + + 1. So that we can override the `execute` method. + The only change we introduce to the method is to explicitly modify xcom relating to `return_values`. + We do this so that the `XComArg` object can work with `expand` function. + + 2. So that we can introduce a keyword argument named `mapper_arr`. + This keyword argument can help as a dummy argument for the `KubernetesPodOperator.partial().expand` method. Any Airflow Operator can be dynamically mapped to runtime artifacts using `Operator.partial(**kwargs).extend(**mapper_kwargs)` post the introduction of [Dynamic Task Mapping](https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html). + The `expand` function takes keyword arguments taken by the operator. + + ## Why override the `execute` method ? + + When we dynamically map vanilla Airflow operators with artifacts generated at runtime, we need to pass that information via `XComArg` to a operator's keyword argument in the `expand` [function](https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#mapping-over-result-of-classic-operators). + The `XComArg` object retrieves XCom values for a particular task based on a `key`, the default key being `return_values`. + Oddly dynamic task mapping [doesn't support XCom values from any other key except](https://github.com/apache/airflow/blob/8a34d25049a060a035d4db4a49cd4a0d0b07fb0b/airflow/models/mappedoperator.py#L150) `return_values` + The values of XCom passed by the `KubernetesPodOperator` are mapped to the `return_values` XCom key. + + The biggest problem this creates is that the values of the Foreach cardinality are stored inside the dictionary of `return_values` and cannot be accessed trivially like : `XComArg(task)['foreach_key']` since they are resolved during runtime. + This puts us in a bind since the only xcom we can retrieve is the full dictionary and we cannot pass that as the iterable for the mapper tasks. + Hence, we inherit the `execute` method and push custom xcom keys (needed by downstream tasks such as metaflow taskids) and modify `return_values` captured from the container whenever a foreach related xcom is passed. + When we encounter a foreach xcom we resolve the cardinality which is passed to an actual list and return that as `return_values`. + This is later useful in the `Workflow.compile` where the operator's `expand` method is called and we are able to retrieve the xcom value. + """ + + template_fields = KubernetesPodOperator.template_fields + ( + "metaflow_pathspec", + "metaflow_run_id", + "metaflow_task_id", + "metaflow_attempt", + "metaflow_step_name", + "metaflow_flow_name", + ) + + def __init__( + self, + *args, + mapper_arr=None, + flow_name=None, + flow_contains_foreach=False, + **kwargs + ) -> None: + super().__init__(*args, **kwargs) + self.mapper_arr = mapper_arr + self._flow_name = flow_name + self._flow_contains_foreach = flow_contains_foreach + self.metaflow_pathspec = AIRFLOW_MACROS.pathspec( + self._flow_name, is_foreach=self._flow_contains_foreach + ) + self.metaflow_run_id = AIRFLOW_MACROS.RUN_ID + self.metaflow_task_id = AIRFLOW_MACROS.create_task_id( + self._flow_contains_foreach + ) + self.metaflow_attempt = AIRFLOW_MACROS.ATTEMPT + self.metaflow_step_name = AIRFLOW_MACROS.STEPNAME + self.metaflow_flow_name = self._flow_name + + def execute(self, context): + result = super().execute(context) + if result is None: + return + ti = context["ti"] + if TASK_ID_XCOM_KEY in result: + ti.xcom_push( + key=TASK_ID_XCOM_KEY, + value=result[TASK_ID_XCOM_KEY], + ) + if FOREACH_CARDINALITY_XCOM_KEY in result: + return list(range(result[FOREACH_CARDINALITY_XCOM_KEY])) + + return MetaflowKubernetesOperator + + +class AirflowTask(object): + def __init__( + self, + name, + operator_type="kubernetes", + flow_name=None, + is_mapper_node=False, + flow_contains_foreach=False, + ): + self.name = name + self._is_mapper_node = is_mapper_node + self._operator_args = None + self._operator_type = operator_type + self._flow_name = flow_name + self._flow_contains_foreach = flow_contains_foreach + + @property + def is_mapper_node(self): + return self._is_mapper_node + + def set_operator_args(self, **kwargs): + self._operator_args = kwargs + return self + + def to_dict(self): + return { + "name": self.name, + "is_mapper_node": self._is_mapper_node, + "operator_type": self._operator_type, + "operator_args": self._operator_args, + } + + @classmethod + def from_dict(cls, task_dict, flow_name=None, flow_contains_foreach=False): + op_args = {} if "operator_args" not in task_dict else task_dict["operator_args"] + is_mapper_node = ( + False if "is_mapper_node" not in task_dict else task_dict["is_mapper_node"] + ) + return cls( + task_dict["name"], + is_mapper_node=is_mapper_node, + operator_type=task_dict["operator_type"] + if "operator_type" in task_dict + else "kubernetes", + flow_name=flow_name, + flow_contains_foreach=flow_contains_foreach, + ).set_operator_args(**op_args) + + def _kubernetes_task(self): + MetaflowKubernetesOperator = get_metaflow_kubernetes_operator() + k8s_args = _kubernetes_pod_operator_args(self._operator_args) + return MetaflowKubernetesOperator( + flow_name=self._flow_name, + flow_contains_foreach=self._flow_contains_foreach, + **k8s_args + ) + + def _kubernetes_mapper_task(self): + MetaflowKubernetesOperator = get_metaflow_kubernetes_operator() + k8s_args = _kubernetes_pod_operator_args(self._operator_args) + return MetaflowKubernetesOperator.partial( + flow_name=self._flow_name, + flow_contains_foreach=self._flow_contains_foreach, + **k8s_args + ) + + def to_task(self): + if self._operator_type == "kubernetes": + if not self.is_mapper_node: + return self._kubernetes_task() + else: + return self._kubernetes_mapper_task() + + +class Workflow(object): + def __init__(self, file_path=None, graph_structure=None, metadata=None, **kwargs): + self._dag_instantiation_params = AirflowDAGArgs(**kwargs) + self._file_path = file_path + self._metadata = metadata + tree = lambda: defaultdict(tree) + self.states = tree() + self.metaflow_params = None + self.graph_structure = graph_structure + + def set_parameters(self, params): + self.metaflow_params = params + + def add_state(self, state): + self.states[state.name] = state + + def to_dict(self): + return dict( + metadata=self._metadata, + graph_structure=self.graph_structure, + states={s: v.to_dict() for s, v in self.states.items()}, + dag_instantiation_params=self._dag_instantiation_params.serialize(), + file_path=self._file_path, + metaflow_params=self.metaflow_params, + ) + + def to_json(self): + return json.dumps(self.to_dict()) + + @classmethod + def from_dict(cls, data_dict): + re_cls = cls( + file_path=data_dict["file_path"], + graph_structure=data_dict["graph_structure"], + metadata=data_dict["metadata"], + ) + re_cls._dag_instantiation_params = AirflowDAGArgs.deserialize( + data_dict["dag_instantiation_params"] + ) + + for sd in data_dict["states"].values(): + re_cls.add_state( + AirflowTask.from_dict(sd, flow_name=data_dict["metadata"]["flow_name"]) + ) + re_cls.set_parameters(data_dict["metaflow_params"]) + return re_cls + + @classmethod + def from_json(cls, json_string): + data = json.loads(json_string) + return cls.from_dict(data) + + def _construct_params(self): + from airflow.models.param import Param + + if self.metaflow_params is None: + return {} + param_dict = {} + for p in self.metaflow_params: + name = p["name"] + del p["name"] + param_dict[name] = Param(**p) + return param_dict + + def compile(self): + from airflow import DAG + + # Airflow 2.0.0 cannot import this, so we have to do it this way. + # `XComArg` is needed for dynamic task mapping and if the airflow installation is of the right + # version (+2.3.0) then the class will be importable. + XComArg = get_xcom_arg_class() + + _validate_minimum_airflow_version() + + if self._metadata["contains_foreach"]: + _validate_dynamic_mapping_compatibility() + # We need to verify if KubernetesPodOperator is of version > 4.2.0 to support foreachs / dynamic task mapping. + # If the dag uses dynamic Task mapping then we throw an error since the `resources` argument in the `KubernetesPodOperator` + # doesn't work for dynamic task mapping for `KubernetesPodOperator` version < 4.2.0. + # For more context check this issue : https://github.com/apache/airflow/issues/24669 + _check_foreach_compatible_kubernetes_provider() + + params_dict = self._construct_params() + # DAG Params can be seen here : + # https://airflow.apache.org/docs/apache-airflow/2.0.0/_api/airflow/models/dag/index.html#airflow.models.dag.DAG + # Airflow 2.0.0 Allows setting Params. + dag = DAG(params=params_dict, **self._dag_instantiation_params.arguments) + dag.fileloc = self._file_path if self._file_path is not None else dag.fileloc + + def add_node(node, parents, dag): + """ + A recursive function to traverse the specialized + graph_structure datastructure. + """ + if type(node) == str: + task = self.states[node].to_task() + if parents: + for parent in parents: + # Handle foreach nodes. + if self.states[node].is_mapper_node: + task = task.expand(mapper_arr=XComArg(parent)) + parent >> task + return [task] # Return Parent + + # this means a split from parent + if type(node) == list: + # this means branching since everything within the list is a list + if all(isinstance(n, list) for n in node): + curr_parents = parents + parent_list = [] + for node_list in node: + last_parent = add_node(node_list, curr_parents, dag) + parent_list.extend(last_parent) + return parent_list + else: + # this means no branching and everything within the list is not a list and can be actual nodes. + curr_parents = parents + for node_x in node: + curr_parents = add_node(node_x, curr_parents, dag) + return curr_parents + + with dag: + parent = None + for node in self.graph_structure: + parent = add_node(node, parent, dag) + + return dag diff --git a/metaflow/plugins/airflow/dag.py b/metaflow/plugins/airflow/dag.py new file mode 100644 index 00000000000..2720fe40e0e --- /dev/null +++ b/metaflow/plugins/airflow/dag.py @@ -0,0 +1,9 @@ +# Deployed on {{deployed_on}} + +CONFIG = {{{config}}} + +{{{utils}}} + +dag = Workflow.from_dict(CONFIG).compile() +with dag: + pass diff --git a/metaflow/plugins/airflow/exception.py b/metaflow/plugins/airflow/exception.py new file mode 100644 index 00000000000..a76a755e22c --- /dev/null +++ b/metaflow/plugins/airflow/exception.py @@ -0,0 +1,12 @@ +from metaflow.exception import MetaflowException + + +class AirflowException(MetaflowException): + headline = "Airflow Exception" + + def __init__(self, msg): + super().__init__(msg) + + +class NotSupportedException(MetaflowException): + headline = "Not yet supported with Airflow" diff --git a/metaflow/plugins/airflow/plumbing/__init__.py b/metaflow/plugins/airflow/plumbing/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/metaflow/plugins/airflow/plumbing/set_parameters.py b/metaflow/plugins/airflow/plumbing/set_parameters.py new file mode 100644 index 00000000000..7a2e4dd3112 --- /dev/null +++ b/metaflow/plugins/airflow/plumbing/set_parameters.py @@ -0,0 +1,21 @@ +import os +import json +import sys + + +def export_parameters(output_file): + input = json.loads(os.environ.get("METAFLOW_PARAMETERS", "{}")) + with open(output_file, "w") as f: + for k in input: + # Replace `-` with `_` is parameter names since `-` isn't an + # allowed character for environment variables. cli.py will + # correctly translate the replaced `-`s. + f.write( + "export METAFLOW_INIT_%s=%s\n" + % (k.upper().replace("-", "_"), json.dumps(input[k])) + ) + os.chmod(output_file, 509) + + +if __name__ == "__main__": + export_parameters(sys.argv[1]) diff --git a/metaflow/plugins/argo/__init__.py b/metaflow/plugins/argo/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/metaflow/plugins/argo/argo_client.py b/metaflow/plugins/argo/argo_client.py new file mode 100644 index 00000000000..bfe70cff5b9 --- /dev/null +++ b/metaflow/plugins/argo/argo_client.py @@ -0,0 +1,182 @@ +import json +import os +import sys + +from metaflow.exception import MetaflowException +from metaflow.plugins.kubernetes.kubernetes_client import KubernetesClient + + +class ArgoClientException(MetaflowException): + headline = "Argo Client error" + + +class ArgoClient(object): + def __init__(self, namespace=None): + + self._kubernetes_client = KubernetesClient() + self._namespace = namespace or "default" + self._group = "argoproj.io" + self._version = "v1alpha1" + + def get_workflow_template(self, name): + client = self._kubernetes_client.get() + try: + return client.CustomObjectsApi().get_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="workflowtemplates", + name=name, + ) + except client.rest.ApiException as e: + if e.status == 404: + return None + raise ArgoClientException( + json.loads(e.body)["message"] if e.body is not None else e.reason + ) + + def register_workflow_template(self, name, workflow_template): + # Unfortunately, Kubernetes client does not handle optimistic + # concurrency control by itself unlike kubectl + client = self._kubernetes_client.get() + try: + workflow_template["metadata"][ + "resourceVersion" + ] = client.CustomObjectsApi().get_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="workflowtemplates", + name=name, + )[ + "metadata" + ][ + "resourceVersion" + ] + except client.rest.ApiException as e: + if e.status == 404: + try: + return client.CustomObjectsApi().create_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="workflowtemplates", + body=workflow_template, + ) + except client.rest.ApiException as e: + raise ArgoClientException( + json.loads(e.body)["message"] + if e.body is not None + else e.reason + ) + else: + raise ArgoClientException( + json.loads(e.body)["message"] if e.body is not None else e.reason + ) + try: + return client.CustomObjectsApi().replace_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="workflowtemplates", + body=workflow_template, + name=name, + ) + except client.rest.ApiException as e: + raise ArgoClientException( + json.loads(e.body)["message"] if e.body is not None else e.reason + ) + + def trigger_workflow_template(self, name, parameters={}): + client = self._kubernetes_client.get() + body = { + "apiVersion": "argoproj.io/v1alpha1", + "kind": "Workflow", + "metadata": {"generateName": name + "-"}, + "spec": { + "workflowTemplateRef": {"name": name}, + "arguments": { + "parameters": [ + {"name": k, "value": json.dumps(v)} + for k, v in parameters.items() + ] + }, + }, + } + try: + return client.CustomObjectsApi().create_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="workflows", + body=body, + ) + except client.rest.ApiException as e: + raise ArgoClientException( + json.loads(e.body)["message"] if e.body is not None else e.reason + ) + + def schedule_workflow_template(self, name, schedule=None): + # Unfortunately, Kubernetes client does not handle optimistic + # concurrency control by itself unlike kubectl + client = self._kubernetes_client.get() + body = { + "apiVersion": "argoproj.io/v1alpha1", + "kind": "CronWorkflow", + "metadata": {"name": name}, + "spec": { + "suspend": schedule is None, + "schedule": schedule, + "workflowSpec": {"workflowTemplateRef": {"name": name}}, + }, + } + try: + body["metadata"][ + "resourceVersion" + ] = client.CustomObjectsApi().get_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="cronworkflows", + name=name, + )[ + "metadata" + ][ + "resourceVersion" + ] + except client.rest.ApiException as e: + # Scheduled workflow does not exist and we want to schedule a workflow + if e.status == 404: + if schedule is None: + return + try: + return client.CustomObjectsApi().create_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="cronworkflows", + body=body, + ) + except client.rest.ApiException as e: + raise ArgoClientException( + json.loads(e.body)["message"] + if e.body is not None + else e.reason + ) + else: + raise ArgoClientException( + json.loads(e.body)["message"] if e.body is not None else e.reason + ) + try: + return client.CustomObjectsApi().replace_namespaced_custom_object( + group=self._group, + version=self._version, + namespace=self._namespace, + plural="cronworkflows", + body=body, + name=name, + ) + except client.rest.ApiException as e: + raise ArgoClientException( + json.loads(e.body)["message"] if e.body is not None else e.reason + ) diff --git a/metaflow/plugins/argo/argo_workflows.py b/metaflow/plugins/argo/argo_workflows.py new file mode 100644 index 00000000000..071e097baf0 --- /dev/null +++ b/metaflow/plugins/argo/argo_workflows.py @@ -0,0 +1,1404 @@ +import json +import os +import shlex +import sys +from collections import defaultdict + +from metaflow import current +from metaflow.decorators import flow_decorators +from metaflow.exception import MetaflowException +from metaflow.metaflow_config import ( + SERVICE_HEADERS, + SERVICE_INTERNAL_URL, + CARD_S3ROOT, + DATASTORE_SYSROOT_S3, + DATATOOLS_S3ROOT, + DEFAULT_METADATA, + KUBERNETES_NAMESPACE, + KUBERNETES_NODE_SELECTOR, + KUBERNETES_SANDBOX_INIT_SCRIPT, + KUBERNETES_SECRETS, + S3_ENDPOINT_URL, + AZURE_STORAGE_BLOB_SERVICE_ENDPOINT, + DATASTORE_SYSROOT_AZURE, + DATASTORE_SYSROOT_GS, + CARD_AZUREROOT, + CARD_GSROOT, +) +from metaflow.mflog import BASH_SAVE_LOGS, bash_capture_logs, export_mflog_env_vars +from metaflow.parameters import deploy_time_eval +from metaflow.util import compress_list, dict_to_cli_options, to_camelcase + +from .argo_client import ArgoClient + + +class ArgoWorkflowsException(MetaflowException): + headline = "Argo Workflows error" + + +class ArgoWorkflowsSchedulingException(MetaflowException): + headline = "Argo Workflows scheduling error" + + +# List of future enhancements - +# 1. Configure Argo metrics. +# 2. Support Argo Events. +# 3. Support resuming failed workflows within Argo Workflows. +# 4. Support gang-scheduled clusters for distributed PyTorch/TF - One option is to +# use volcano - https://github.com/volcano-sh/volcano/tree/master/example/integrations/argo +# 5. Support GitOps workflows. +# 6. Add Metaflow tags to labels/annotations. +# 7. Support Multi-cluster scheduling - https://github.com/argoproj/argo-workflows/issues/3523#issuecomment-792307297 +# 8. Support for workflow notifications. +# 9. Support R lang. +# 10.Ping @savin at slack.outerbounds.co for any feature request. + + +class ArgoWorkflows(object): + def __init__( + self, + name, + graph, + flow, + code_package_sha, + code_package_url, + production_token, + metadata, + flow_datastore, + environment, + event_logger, + monitor, + tags=None, + namespace=None, + username=None, + max_workers=None, + workflow_timeout=None, + workflow_priority=None, + ): + # Some high-level notes - + # + # Fail-fast behavior for Argo Workflows - Argo stops + # scheduling new steps as soon as it detects that one of the DAG nodes + # has failed. After waiting for all the scheduled DAG nodes to run till + # completion, Argo with fail the DAG. This implies that after a node + # has failed, it may be awhile before the entire DAG is marked as + # failed. There is nothing Metaflow can do here for failing even + # faster (as of Argo 3.2). + # + # argo stop` vs `argo terminate` - since we don't currently + # rely on any exit handlers, it's safe to either stop or terminate any running + # argo workflow deployed through Metaflow. This may not hold true, once we + # integrate with Argo Events. + # + # Currently, an Argo Workflow can only execute entirely within a single + # Kubernetes namespace. Multi-cluster / Multi-namespace execution is on the + # deck for v3.4 release for Argo Workflows; beyond which point, we will be + # able to support them natively. + # + # Since this implementation generates numerous templates on the fly, please + # ensure that your Argo Workflows controller doesn't restrict + # templateReferencing. + + self.name = name + self.graph = graph + self.flow = flow + self.code_package_sha = code_package_sha + self.code_package_url = code_package_url + self.production_token = production_token + self.metadata = metadata + self.flow_datastore = flow_datastore + self.environment = environment + self.event_logger = event_logger + self.monitor = monitor + self.tags = tags + self.namespace = namespace + self.username = username + self.max_workers = max_workers + self.workflow_timeout = workflow_timeout + self.workflow_priority = workflow_priority + + self.parameters = self._process_parameters() + self._workflow_template = self._compile() + self._cron = self._cron() + + def __str__(self): + return str(self._workflow_template) + + def deploy(self): + try: + ArgoClient(namespace=KUBERNETES_NAMESPACE).register_workflow_template( + self.name, self._workflow_template.to_json() + ) + except Exception as e: + raise ArgoWorkflowsException(str(e)) + + @staticmethod + def _sanitize(name): + # Metaflow allows underscores in node names, which are disallowed in Argo + # Workflow template names - so we swap them with hyphens which are not + # allowed by Metaflow - guaranteeing uniqueness. + return name.replace("_", "-") + + @classmethod + def trigger(cls, name, parameters=None): + if parameters is None: + parameters = {} + try: + workflow_template = ArgoClient( + namespace=KUBERNETES_NAMESPACE + ).get_workflow_template(name) + except Exception as e: + raise ArgoWorkflowsException(str(e)) + if workflow_template is None: + raise ArgoWorkflowsException( + "The workflow *%s* doesn't exist on Argo Workflows in namespace *%s*. " + "Please deploy your flow first." % (name, KUBERNETES_NAMESPACE) + ) + else: + try: + # Check that the workflow was deployed through Metaflow + workflow_template["metadata"]["annotations"]["metaflow/owner"] + except KeyError as e: + raise ArgoWorkflowsException( + "An existing non-metaflow workflow with the same name as " + "*%s* already exists in Argo Workflows. \nPlease modify the " + "name of this flow or delete your existing workflow on Argo " + "Workflows before proceeding." % name + ) + try: + return ArgoClient(namespace=KUBERNETES_NAMESPACE).trigger_workflow_template( + name, parameters + ) + except Exception as e: + raise ArgoWorkflowsException(str(e)) + + def _cron(self): + schedule = self.flow._flow_decorators.get("schedule") + if schedule: + # Remove the field "Year" if it exists + return " ".join(schedule.schedule.split()[:5]) + return None + + def schedule(self): + try: + ArgoClient(namespace=KUBERNETES_NAMESPACE).schedule_workflow_template( + self.name, self._cron + ) + except Exception as e: + raise ArgoWorkflowsSchedulingException(str(e)) + + def trigger_explanation(self): + if self._cron: + return ( + "This workflow triggers automatically via the CronWorkflow *%s*." + % self.name + ) + else: + return "No triggers defined. You need to launch this workflow manually." + + @classmethod + def get_existing_deployment(cls, name): + workflow_template = ArgoClient( + namespace=KUBERNETES_NAMESPACE + ).get_workflow_template(name) + if workflow_template is not None: + try: + return ( + workflow_template["metadata"]["annotations"]["metaflow/owner"], + workflow_template["metadata"]["annotations"][ + "metaflow/production_token" + ], + ) + except KeyError as e: + raise ArgoWorkflowsException( + "An existing non-metaflow workflow with the same name as " + "*%s* already exists in Argo Workflows. \nPlease modify the " + "name of this flow or delete your existing workflow on Argo " + "Workflows before proceeding." % name + ) + return None + + def _process_parameters(self): + parameters = [] + has_schedule = self._cron() is not None + seen = set() + for var, param in self.flow._get_parameters(): + # Throw an exception if the parameter is specified twice. + norm = param.name.lower() + if norm in seen: + raise MetaflowException( + "Parameter *%s* is specified twice. " + "Note that parameter names are " + "case-insensitive." % param.name + ) + seen.add(norm) + + is_required = param.kwargs.get("required", False) + # Throw an exception if a schedule is set for a flow with required + # parameters with no defaults. We currently don't have any notion + # of data triggers in Argo Workflows. + + # TODO: Support Argo Events for data triggering in the near future. + if "default" not in param.kwargs and is_required and has_schedule: + raise MetaflowException( + "The parameter *%s* does not have a default and is required. " + "Scheduling such parameters via Argo CronWorkflows is not " + "currently supported." % param.name + ) + value = deploy_time_eval(param.kwargs.get("default")) + # If the value is not required and the value is None, we set the value to + # the JSON equivalent of None to please argo-workflows. + if not is_required or value is not None: + value = json.dumps(value) + parameters.append( + dict(name=param.name, value=value, description=param.kwargs.get("help")) + ) + return parameters + + def _compile(self): + # This method compiles a Metaflow FlowSpec into Argo WorkflowTemplate + # + # WorkflowTemplate + # | + # -- WorkflowSpec + # | + # -- Array