Bump follow-redirects from 1.14.7 to 1.14.9 in /metaflow/plugins/cards/ui #966
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=60&v=4){kind=small}
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.7...v1.14.9) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>
dependabot
bot
added
the
dependencies
savingoyal
approved these changes
Feb 21, 2022
savingoyal
deleted the
dependabot/npm_and_yarn/metaflow/plugins/cards/ui/follow-redirects-1.14.9
branch
npow
added a commit
to npow/metaflow
that referenced
this pull request
Jan 12, 2023
* Don't explicitly break py2 support (#962) * Don't explicitly break py2 support * Typo * Typo * Pass the paths used by the external interpreter to the one launching the Env Escape server (#960) We were currently not reflecting possible values passed using PYTHONPATH or programmatically added to the interpreter launching the environment escape server. * Bump follow-redirects in /metaflow/plugins/cards/ui (#966) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.7...v1.14.9) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Adding export of graph to JSON for cli (#955) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS (#972) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS * Ran black * emit app tag for AWS Batch jobs (#970) * Bump to 2.5.3 (#974) * Extension packaging improvements (#959) * Fixed dual distribution import in some cases; fixed get_plugin_cli extension If sys.path contains the same directory multiple times, metadata.distributions() will list a package multiple times which breaks the import mechanism we have. This is addressed by skipping duplicate packages. get_plugin_cli for extensions wasn't evaluated late enough therefore not allowing additional parameters to be added. * Fix issue with metaflow_extensions and Conda environment In a Conda environment, it is not possible to re-resolve all the metaflow_extension packages since they appear as a single directory. In that case, we resolve on the INFO file to give us the proper information. Also fixed the way metaflow_extensions are added to the Conda environment to avoid leaking more information than needed. * Validate configuration files for extensions and de-duplicate for internal ns packages * Update packaging of metaflow_extensions packages Packaging is now handled directly by extension_support.py and each distribution/package can define its own suffixes to include. * Rework the import mechanism for modules contained in metaflow_extensions This change does away with the deprecated load_module and also improves handling of loading children modules and _orig modules * Address comments * Squash merge of origin/master * Fix issue when packages were left out * Merge get_pinned_conda_libs * Fix issue with __init__.py in non-distribution packages * Moving card import changes on top of Romain's branch (#973) * Moving card import changes on top of Romain's branch - tiny bug fix to extensions. - Added card related refactor to support mfextinit and regular package. - removing card packaging related code in decorator + everywhere else. * Tiny bug fix on top of romain's new changes. * removing unneccessary logic * Added a bit more inline documentation and addressed documentation comments * Properly handle parsing of package requirements * Fix Pylint errors in some case of metaflow_extensions module aliasing * Properly handle case of files at the root of PYTHONPATH package * Tests for Extensions (#978) * Added tests for extensions. Checking if they work. * dummy commit to see if things work. * fix * debug * bug fix. * possible fix. * tweeking context * Bug fix to tests. * dummy commit * bug fix * Added extension test to core tests. - remove seperate gh action * removing files. * fix * added extension test to py3 context - remove redundant complexity. Co-authored-by: Valay Dave <[email protected]> * pass DEFAULT_AWS_CLIENT_PROVIDER to remote tasks (#982) * Fixing some hacky plumbing in card test suite (#967) * Fixing some hacky plumbing for card tests. - Added `--file` arg to card list cli - Using files to get the information written by cli - changes to test to use files instead of stdout - Piping stderrors to stderrr to capture card not found errors. * Removing `\n` from all card tests assertions - `\n` was there because earlier we read stdout. - Now since we read files, it will not be needed. * Simplify mflog (#979) * Fix extension root determination in some cases + card tweaks (#989) * Use importlib_metadata 2.8.3 for Python 3.4, 3.5 but 4.8.3 for Python 3.6+ (#988) * Use importlib_metadata 2.8.3 for Python 3.4 and 3.5 but 4.8.3 for Python 3.6+ This is required because in importlib_metadata 3.4.0, a field called `_normalized_name` was introduced and is relied on by later versions of importlib_metadata. When importlib_metadata looks for distributions, it queries all registered/installed importlib_metadata and if an older version (like 2.8.3) returns something that does not have this field, it causes a crash. * Add __init__.py files to make the vendored package non namespace * Allow configuring the root directory where artifacts go when pulling from S3. (#991) Co-authored-by: kgullikson <[email protected]> * Add an option to pass a `role` argument to S3() to provide a role (#987) * Add an option to pass a `role` argument to S3() to provide a role for the S3 client * Move to partial to be able to pickle ops * Address comments * bump version to 2.5.4 (#993) * Bug fix for datetime index in default card (#981) * Bug fix for datetime_indexes. * ran black * Bump minimist from 1.2.5 to 1.2.6 in /metaflow/plugins/cards/ui (#995) Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6. - [Release notes](https://github.com/substack/minimist/releases) - [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6) --- updated-dependencies: - dependency-name: minimist dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix black - set upper bound for click version (#1006) * Add tags to `current` singleton (#1019) * first attempt * fix _set_env param typo * add assertions for tags to current singleton test * Dispatch Metaflow flows to Argo Workflows (#992) * Dispatch Metaflow flows to Argo Workflows * Remove spurious files * Make black happy * Make black happy again * Remove spurious TODOs * Handle non-numeric task ids * Fix typos * don't actively automount service account tokens * make black happy * Update @kubernetes * remove spurious commits * fix issue with hyphens * raise memory for tests * change limits to requests * pin back test resources * update refresh timeout * stylistic nits * Add lint check * make black happy * support gpus * drop extra bracket * fix * Skip sleep if not retryting S3 operations (#1001) * Don't sleep if not retrying S3 operations * Remove unused env variable * VSCode is too smart * Bump minor version for release (#1022) * Alternate way of adding metadata to cloned tasks (#1003) * Alternate way of adding metadata to cloned tasks This adds it on the client side * Clean up metadata passing. Also optimize use within the client * Typo * Fix black test * Keep black happy * Added tests * Ignore namespace when getting origin task * Bump cross-fetch from 3.1.4 to 3.1.5 in /metaflow/plugins/cards/ui (#1027) Bumps [cross-fetch](https://github.com/lquixada/cross-fetch) from 3.1.4 to 3.1.5. - [Release notes](https://github.com/lquixada/cross-fetch/releases) - [Commits](https://github.com/lquixada/cross-fetch/compare/v3.1.4...v3.1.5) --- updated-dependencies: - dependency-name: cross-fetch dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed bug from #1023 (#1025) * current.pathspec to return None when used outside Flow (#1033) * current.pathspec to return None when used outside Flow * formatted with black * Fixing bug in the `card list` command (#1044) - echo_always defaulted to stderr :( * Enable MinIO/EMC-ECS as blob stores for Metaflow (#1045) * Enable EMCECS * Enable MinIO & EMC-ECS blob stores for Metaflow * fix InvalidRange issue * [tag-mutation-project] Stash system info in task metadata (#1039) * stash system info as metadata (in addition to system_tags today) * improve * avoid parsing on ":" for user id * remove date - it's not well defined and created_at is available * python_version is in metadata now, and can vary between orig and resume runs * address CR comments * add the comments * Address issues with S3 get and ranges (#1034) * Address issues with S3 get and ranges This addresses two issues: - get and get_many would not properly populate the range_info in S3Object - get_many on different ranges of the same file would return incorrect results * Add one more test (merge with Netflix internal tests) * Put Run() test behind a check * Address comments * Fix _new_task calling bug in LocalMetadataProvider (#1046) * Release 2.6.1 (#1047) * Support default secrets (#1048) * Run id file should be written prior to execution when resuming (#1051) * Run id file should be written prior to execution when resuming * separate run id test so we can exclude from batch * release 2.6.2 (#1052) * Fix instance metadata calls for IMDSV2 (#1053) * bump version to 2.6.3 (#1054) * Small reorder of imports to make things more consistent (#1055) * [tag-mutation-project] Local metadata provider - object read paths to return ancestral Run tags (#1043) * wip * Fix nits * typo fix. (#1068) * This is the Jackie's tag changes with an additional test and a fix. (#1063) * rebase on master * add replace_many as candidate subcommand * updates from some CR suggestions * propagate opt rename * more changes follow CR * improve consistency guarantees of local metadata tag mutations (retries if a race was lost) * write comment * use replace-many logic to back "tag replace" * finalize client API for tag operations * fix retry logic * Add task.tags == run.tags check in test * [tag-mutation-project] Add tag mutation support to ServiceMetadataProvider * fix version * Enable TagMutationTest in batch and k8s * Bogus commit * minor UX fixes on CLI * Fully support CliCheck for TagMutationTest * add tag and tag set validation to local metadata code path Validate tags on run / resume paths too * flat nor flatten on CLI * Logs really do come via stdout - related to CliCheck refactor * argo and step functions - validate tags as soon as possible (before other user messaging) * no capture_output pre 3.7 * Add another test for tag mutation * Properly pass down the metadata provider There were two cases when the metadata provider was not properly passed down to the client: - when calling the client within a flow - when invoking the `tag` command This fixes both issues and also allows the user to introspect the metadata provider and datastore being used for the flow * Do not make public metadata and datastore in current object * Typo Co-authored-by: Jackie Tung <[email protected]> Co-authored-by: jackie-ob <[email protected]> * Add two options to resume: the ability to specify a run-id and the ab… (#1059) * Add two options to resume: the ability to specify a run-id and the ability to only clone tasks The scenario for this use is mainly for external schedulers that are capable of "resuming" failed runs by only re-executing the ones that failed or didn't execute the first time. We need a way for Metaflow to "clone" the tasks that are not re-executed. * Small formatting fix * Made clone-only resume possibly re-entrant * Addressed comments * Clarify the reentrant behavior of resume * Improve sidecar message handling (#1057) * Improve sidecar message handling Several changes: - allow side-passing of a static "context" to the sidecar process to keep messages as small as possible - clean up the interfaces and make it clearer what is what (ie: there is an emitter inside the process and then a sidecar that runs outside) - better error checking - better handling of shutdown (sidecars will now properly get a shutdown message giving them the opportunity to clean up -- this may add a bit of latency at the end of the program execution. Also improved message when plugins don't load in case of an issue with Metaflow extensions * Add the possibility to pass additional context to sidecars This allows more information to be provided after the sidecar is created but still maintains small message sizes * Better handling of initial context message for sidecars In the previous implementation, a failure to send the initial context message could crash the flow. This change properly handles that message like any other message (so has no impact on the flow itself). It also implements better retry policies to send the context. Finally, this change also improves error handling: if a message fails to send, in the previous implementation, it was likely that the next successful one would be an invalid json. * Improve handling of corner conditions on message send * Restart sidecar faster when sidecar process dies * Properly send context after sidecar restart * Remove the specific context from monitors/loggers Subclasses can now handle contexts directly in the subclass. Refactoring also cleaned up what a sidecar and a sidecar worker do by refactoring some of the code out. * Typo: fix names * Minor typos for sidecar changes * Clean up sidecars * More cleanups for sidecars * Addressed final comments for sidecars * Small typo in NullEventLogger.send (#1071) Pushing directly per our conversation with Savin * Bump version to 2.7.0 * Change behavior of env escape when module is present (#1072) When a module configured in the env escape was present in the local environment, we would raise an error. We now print a message and use the module present in the environment (not the environment escape). This allows the env escape modules to provide a "fallback" in case the module is not present but to still use the local module if present * Bump version to 2.7.1 (#1073) * Fix an issue with the environment escape server directory (#1074) The environment escape server would launch in the cwd. This could cause issues if, for example, a `metaflow` directory existed in that cwd (because it would then take precedence over the actual metaflow. To remediate this situation, the server is launched in the configuration directory (where we know what there is). Also took care of a quick annoyance of having to create an empty overrides.py file for everything. This is now no longer required * Support session_vars and client_params in S3() (#1069) * Support session_vars and client_params in S3() This allows the user to pass down session variables and client parameters to set when getting a S3 client. * Addressed comments about default configuration * Support M1 Macs (#1077) * changes to @kubernetes for sandbox (#1079) * Changed error to warning in AWS retry (#1081) Co-authored-by: Preetam Joshi <[email protected]> * bump to 2.7.2 (#1083) * Fixed s3util test (#1084) * Fixing s3util test caset * added comment Co-authored-by: Preetam Joshi <[email protected]> * Bump svelte from 3.46.1 to 3.49.0 in /metaflow/plugins/cards/ui (#1086) Bumps [svelte](https://github.com/sveltejs/svelte) from 3.46.1 to 3.49.0. - [Release notes](https://github.com/sveltejs/svelte/releases) - [Changelog](https://github.com/sveltejs/svelte/blob/master/CHANGELOG.md) - [Commits](https://github.com/sveltejs/svelte/compare/v3.46.1...v3.49.0) --- updated-dependencies: - dependency-name: svelte dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Metadata version check flag (#1088) * Bump terser from 5.10.0 to 5.14.2 in /metaflow/plugins/cards/ui (#1090) Bumps [terser](https://github.com/terser/terser) from 5.10.0 to 5.14.2. - [Release notes](https://github.com/terser/terser/releases) - [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md) - [Commits](https://github.com/terser/terser/commits) --- updated-dependencies: - dependency-name: terser dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix r CI deps (#1092) * fix r CI deps * fix fractional resoure handling for batch (#1089) * bump version (#1093) * Fix docstrings for the API reference (no functional changes!) (#1076) Make the docstring format in public APIs compliant with the new API reference framework * Move a sys.path modification in s3op to __main__ (#1095) In its current location, this could modify the sys.path of the current running metaflow which could have nefarious consequences with the escape hatch which uses sys.path to determine the outside environment's python path. The following scenario would cause issues: - metaflow is installed in the usual path on the system - a conda environment was manually bootstrapped from a directory A. + at this point, sys.path starts with `A` and then contains the other system includes + s3op.py is imported at some point by the Conda installer when it calls `get_many` + this modifies sys.path to insert, at the beginning, the parent of Metaflow; so in this case, sys.path looks something like ['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...] + when the escape hatch trampolines are created, this sys.path is used to determine what the sys.path for the outside interpreter is. + in A, we create: * INFO * metaflow * metaflow_extensions which properly describe the installation of metaflow - when the escape hatch client runs, it runs in the conda environment and uses metaflow created in A. - when the client wants to start the server, this is where we run into issues because, at this point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages' in which it will find metaflow. It will therefore use that metaflow (which is the same as the one linked in A) to start the server. This runs into issues though because A is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`. This will cause issues if multiple extensions are installed there (it will complain about duplicate configurations for example. The `INFO` file typically used to solve this problem is not read as it was not present for the TL metaflow. This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting sys.path when the module is simply included (and not called as a script). * Airflow Support (#1094) * Airflow on Kubernetes minus Foreachs. - Support for all metaflow construct without foreach and sensors Squashed commit of the following: commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1 Author: Valay Dave <[email protected]> Date: Fri Jul 29 01:06:26 2022 +0000 Removed sernsors and banned foreach's commit 8d517c4fecc6568777ad03eca81aaacfa3e91156 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:59:01 2022 +0000 commiting k8s related file from master. commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:54:45 2022 +0000 Uncommented code for foreach support with k8s KubernetesPodOperator version 4.2.0 renamed `resources` to `container_resources` - Check : (https://github.com/apache/airflow/pull/24673) / - (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85) This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator` and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. If it is > 4.2.0 then we set the argument as `container_resources` The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future. So we will only use it for `KuberentesPodOperator` version < 4.2.0 The `resources` argument will also not work for foreach's. commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:31:58 2022 +0000 nit fixes : - fixing comments. - refactor some variable/function names. commit 2079293fbba0d3d862476a7d67b36af8a3389342 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:14:53 2022 +0000 change `token` to `production_token` commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:11:56 2022 +0000 Refactored import Airflow Sensors. commit b1472d5f7a629024ca45e8b83700400d02a4d455 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:08:41 2022 +0000 new comment on `startup_timeout_seconds` env var. commit 6d81b758e8f06911258d26f790029f557488a0d7 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:06:09 2022 +0000 Removing traces of `@airflow_schedule_interval` commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946 Author: Valay Dave <[email protected]> Date: Thu Jul 14 12:43:08 2022 -0700 Foreach polish (valayDave/metaflow#62) * Removing unused imports * Added validation logic for airflow version numbers with foreaches * Removed `airflow_schedule_interval` decorator. * Added production/deployment token related changes - Uses s3 as a backend to store the production token - Token used for avoiding nameclashes - token stored via `FlowDatastore` * Graph type validation for airflow foreachs - Airflow foreachs only support single node fanout. - validation invalidates graphs with nested foreachs * Added configuration about startup_timeout. * Added final todo on `resources` argument of k8sOp - added a commented code block - it needs to be uncommented when airflow releasese the patch for the op - Code seems feature complete keeping aside airflow patch commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9 Author: Valay Dave <[email protected]> Date: Thu Jul 7 19:33:07 2022 +0000 Removed retries from user-defaults. commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43 Author: Valay Dave <[email protected]> Date: Wed Jul 6 16:29:33 2022 +0000 updated pod startup time commit fce2bd263f368dbb78a34ac71f64e13c89277222 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:11 2022 +0000 Adding default 1 retry for any airflow worker. commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321 Author: Valay Dave <[email protected]> Date: Mon Jun 27 01:22:42 2022 +0000 Airflow Foreach Integration - Simple one node foreach-join support as gaurenteed by airflow - Fixed env variable setting issue - introduced MetaflowKuberentesOperator - Created a new operator to allow smootness in plumbing xcom values - Some todos commit d319fa915c558d82f1d127736ce34d3ae0da521d Author: Valay Dave <[email protected]> Date: Fri Jun 24 21:12:09 2022 +0000 simplifying run-id macro. commit 0ffc813b1c4e6ba0103be51520f42d191371741a Author: Valay Dave <[email protected]> Date: Fri Jun 24 11:51:42 2022 -0700 Refactored parameter macro settings. (valayDave/metaflow#60) commit a3a495077f34183d706c0edbe56d6213766bf5f6 Author: Valay Dave <[email protected]> Date: Fri Jun 24 02:05:57 2022 +0000 added comment on need for `start_date` commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec Author: Valay Dave <[email protected]> Date: Tue Jun 21 06:03:56 2022 +0000 Refactored an `id_creator` method. commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:52:05 2022 +0000 refactor : -`RUN_ID_LEN` to `RUN_HASH_ID_LEN` - `TASK_ID_LEN` to `TASK_ID_HASH_LEN` commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2 Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:48:55 2022 +0000 refactored an error string commit 11458188b6c59d044fca0dd2d1f5024ec84f6488 Author: Valay Dave <[email protected]> Date: Mon Jun 20 22:42:36 2022 -0700 addressing savins comments. (#59) - Added many adhoc changes based for some comments. - Integrated secrets and `KUBERNETES_SECRETS` - cleaned up parameter setting - cleaned up setting of scheduling interval - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID` - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate` - Checking if dagfile and flow file are same. - fixing variable names. - checking out `kubernetes_decorator.py` from master (6441ed5) - bug fixing secret setting in airflow. - simplified parameter type parsing logic - refactoring airflow argument parsing code. commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646 Author: Valay Dave <[email protected]> Date: Mon Jun 13 14:02:57 2022 -0700 Addressing Final comments. (#57) - Added dag-run timeout. - airflow related scheduling checks in decorator. - Auto naming sensors if no name is provided - Annotations to k8s operators - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`) - annotation bug fix - setting`workflow-timeout` for only scheduled dags commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:50:49 2022 +0000 k8s bug fix commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:39:50 2022 +0000 removed un-used function commit 70e285e9a7cfbec71fc293508a62c96f33562a01 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:38:37 2022 +0000 Removed unused `sanitize_label` function commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:37:34 2022 +0000 GPU support added + container naming same as argo commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:25:17 2022 +0000 Refactored sensors to different files + bug fix - bug caused due `util.compress_list`. - The function doesn't play nice with strings with variety of characters. - Ensured that exceptions are handled appropriately. - Made new file for each sensor under `airflow.sensors` module. commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673 Author: Valay Dave <[email protected]> Date: Sat Jun 4 01:41:49 2022 +0000 ran black. commit 558c82f65b383ed0d61ded6bc80326471e284550 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:32:48 2022 -0700 Moving information from airflow_utils to compiler (#56) - commenting todos to organize unfinished changes. - some environment variables set via`V1EnvVar` - `client.V1ObjectFieldSelector` mapped env vars were not working in json form - Moving k8s operator import into its own function. - env vars moved. commit 9bb5f638792a671164ec95891e97f599e9a3385f Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:06:03 2022 +0000 added mising Run-id prefixes to variables. - merged `hash` and `dash_connect` filters. commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:00:22 2022 +0000 nit fix : variable name change. commit 660756f952ebd92ba1e26d7f908b81036c31ff10 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:58:34 2022 +0000 nit fixes to dag.py's templating variables. commit 1202f5bc92f76df52b5957f11c8574cadfa62196 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:56:53 2022 +0000 Fixed defaults passing - Addressed comments for airflow.py commit b9387dd428c1a37f9a3bfe2c72cab475da708c02 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:52:24 2022 +0000 Following Changes: - Refactors setting scheduling interval - refactor dag file creating function - refactored is_active to is_paused_upon_creation - removed catchup commit 054e3f389febc6c447494a1dedb01228f5f5650f Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:33:25 2022 +0000 Multiple Changes based on comments: 1. refactored `create_k8s_args` into _to_job 2. Addressed comments for snake casing 3. refactored `attrs` for simplicity. 4. refactored `metaflow_parameters` to `parameters`. 5. Refactored setting of `input_paths` commit d481b2fca7914b6b657a69af407cfe1a894a46dc Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:42:24 2022 +0000 Removed Sensor metadata extraction. commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3 Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:30:34 2022 +0000 porting savin's comments - next changes : addressing comments. commit 3f2353a647e53bc240e28792769c42a71ea8f8c9 Merge: d370ffb c1ff469 Author: Valay Dave <[email protected]> Date: Thu Jul 28 23:52:16 2022 +0000 Merge branch 'master' into airflow commit d370ffb248411ad4675f9d55de709dbd75d3806e Merge: a82f144 e4eb751 Author: Valay Dave <[email protected]> Date: Thu Jul 14 19:38:48 2022 +0000 Merge branch 'master' into airflow commit a82f1447b414171fc5611758cb6c12fc692f55f9 Merge: bdb1f0d 6f097e3 Author: Valay Dave <[email protected]> Date: Wed Jul 13 00:35:49 2022 +0000 Merge branch 'master' into airflow commit bdb1f0dd248d01318d4a493c75b6f54248c7be64 Merge: 8511215 f9a4968 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:51 2022 +0000 Merge branch 'master' into airflow commit 85112158cd352cb7de95a2262c011c6f43d98283 Author: Valay Dave <[email protected]> Date: Tue Jun 21 02:53:11 2022 +0000 Bug fix from master merge. commit 90c06f12bb14eda51c6a641766c5f67d6763abaa Merge: 0fb73af 6441ed5 Author: Valay Dave <[email protected]> Date: Mon Jun 20 21:20:20 2022 +0000 Merge branch 'master' into airflow commit 0fb73af8af9fca2875261e3bdd305a0daab1b229 Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:53:10 2022 +0000 squashing bugs after changes from master. commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d Merge: 7bdf662 ffff49b Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:20:38 2022 +0000 Merge branch 'master' into af-mmr commit 7bdf662e14966b929b8369c65d5bd3bbe5741937 Author: Valay Dave <[email protected]> Date: Mon May 16 17:42:38 2022 -0700 Airflow sensor api (#3) * Fixed run-id setting - Change gaurentees that multiple dags triggered at same moment have unique run-id * added allow multiple in `Decorator` class * Airflow sensor integration. >> support added for : - ExternalTaskSensor - S3KeySensor - SqlSensor >> sensors allow multiple decorators >> sensors accept those arguments which are supported by airflow * Added `@airflow_schedule_interval` decorator * Fixing bug run-id related in env variable setting. commit 2604a29452e794354cf4c612f48bae7cf45856ee Author: Valay Dave <[email protected]> Date: Thu Apr 21 18:26:59 2022 +0000 Addressed comments. commit 584e88b679fed7d6eec8ce564bf3707359170568 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:33:55 2022 +0000 fixed printing bug commit 169ac1535e5567149d94749ddaf70264e882d62c Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:30:59 2022 +0000 Option help bug fix. commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:25:54 2022 +0000 variable renamemetaflow_specific_args commit 0c779abcd1d9574878da6de8183461b53e0da366 Merge: d299b13 5a61508 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:23:10 2022 +0000 Merge branch 'airflow-tests' into airflow commit 5a61508e61583b567ef8d3fea04e049d74a6d973 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:22:54 2022 +0000 Removing un-used code / resolved-todos. commit d030830f2543f489a1c4ebd17da1b47942f041d6 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:06:03 2022 +0000 ran black, commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45 Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:04:19 2022 +0000 Merge branch 'master' into airflow-tests commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:02:37 2022 +0000 Merge branch 'master' into airflow commit f2cb3197725f11520da0d49cbeef8de215c243eb Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:54:03 2022 +0000 reverting change. commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4 Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:47:41 2022 +0000 3 changes: - Removing s3 dep - remove uesless import - added `deployed_on` in dag file template commit c6afba95f5ec05acf7f33fd3228cffd784556e3b Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:50:52 2022 +0000 Fixed passing secrets with kubernetes. commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:04:22 2022 +0000 Refactored code . - removed compute/k8s.py - Moved k8s code to airflow_compiler.py - ran isort to airflow_compiler.py commit d1c343dbbffbddbebd2aeda26d6846e595144e0b Author: Valay Dave <[email protected]> Date: Fri Apr 15 18:02:25 2022 +0000 Added validations about: - un-supported decorators - foreach Changed where validations are done to not save the package. commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce Author: Valay Dave <[email protected]> Date: Fri Apr 15 03:34:26 2022 +0000 Fixing mf log related bug - No double logging on metaflow. commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:10:51 2022 +0000 Removed usless code WRT project decorator. commit 5ad9a3949e351b0ac13f11df13446953932e8ffc Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:03:19 2022 +0000 Remove readme. commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:02:38 2022 +0000 Made file path required arguement. commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:01:07 2022 +0000 changed `--is-active`->`--is-paused-upon-creation` - dags are active by default. commit 5b98f937a62ee74de8aed8b0efde5045a28f068b Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:55:46 2022 +0000 shortened length of run-id and task-id hashes. commit e53426eaa4b156e8bd70ae7510c2e7c66745d101 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:41:32 2022 +0000 Removing un-used args. commit 72cbbfc7424f9be415c22d9144b16a0953f15295 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:39:59 2022 +0000 Moved exceptions to airflow compiler commit b2970ddaa86c393c8abb7f203f6507c386ecbe00 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:33:02 2022 +0000 Changes based on PR comments: - removed airflow xcom push file , moved to decorator code - removed prefix configuration - nit fixes. commit 9e622bac5a75eb9e7a6594d8fa0e47f076634b44 Author: Valay Dave <[email protected]> Date: Mon Apr 11 20:39:00 2022 +0000 Removing un-used code paths + code cleanup commit 7425f62cff2c9128eea785223ddeb40fa2d8f503 Author: Valay Dave <[email protected]> Date: Mon Apr 11 19:45:04 2022 +0000 Fixing bug fix in schedule. commit eb775cbadd1d2d2c90f160a95a0f42c8ff0d7f4c Author: Valay Dave <[email protected]> Date: Sun Apr 10 02:52:59 2022 +0000 Bug fixes WRT Kubernetes secrets + k8s deployments. - Fixing some error messages. - Added some comments. commit 04c92b92c312a4789d3c1e156f61ef57b08dba9f Author: Valay Dave <[email protected]> Date: Sun Apr 10 01:20:53 2022 +0000 Added secrets support. commit 4a0a85dff77327640233767e567aee2b379ac13e Author: Valay Dave <[email protected]> Date: Sun Apr 10 00:11:46 2022 +0000 Bug fix. commit af91099c0a30c26b58d58696a3ef697ec49a8503 Author: Valay Dave <[email protected]> Date: Sun Apr 10 00:03:34 2022 +0000 bug fix. commit c17f04a253dfe6118e2779db79da9669aa2fcef2 Author: Valay Dave <[email protected]> Date: Sat Apr 9 23:55:41 2022 +0000 Bug fix in active defaults. commit 0d372361297857076df6af235d1de7005ac1544a Author: Valay Dave <[email protected]> Date: Sat Apr 9 23:54:02 2022 +0000 @project, @schedule, default active dag support. - Added a flag to allow setting dag as active on creation - Airflow compatible schedule interval - Project name fixes. commit 5c97b15cb11b5e8279befc5b14c239463750e9b7 Author: Valay Dave <[email protected]> Date: Thu Apr 7 21:15:18 2022 +0000 Max workers and worker pool support. commit 9c973f2f44c3cb3a98e3e63f6e4dcef898bc8bf2 Author: Valay Dave <[email protected]> Date: Thu Apr 7 19:34:33 2022 +0000 Adding exceptions for missing features. commit 2a946e2f083a34b4b6ed84c70aebf96b084ee8a2 Author: Valay Dave <[email protected]> Date: Mon Mar 28 19:34:11 2022 +0000 2 changes : - removed hacky line - added support to directly throw dags in s3. commit e0772ec1bad473482c6fd19f8c5e8b9845303c0a Author: Valay Dave <[email protected]> Date: Wed Mar 23 22:38:20 2022 +0000 fixing bugs in service account setting commit 874b94aeeabc664f12551864eff9d8fdc24dc37b Author: Valay Dave <[email protected]> Date: Sun Mar 20 23:49:15 2022 +0000 Added support for Branching with Airflow - remove `next` function in `AirflowTask` - `AirflowTask`s has no knowledge of next tasks. - removed todos + added some todos - Graph construction on airflow side using graph_structure datastructure. - graph_structure comes from`FlowGraph.output_steps()[1]` commit 8e9f649bd8c51171c38a1e5af70a44a85e7009ca Author: Valay Dave <[email protected]> Date: Sun Mar 20 02:33:04 2022 +0000 Added hacky line commit fd5db04cf0a81b14efda5eaf40cd9227e2bac0d3 Author: Valay Dave <[email protected]> Date: Sun Mar 20 02:06:38 2022 +0000 Removed hacky line. commit 5b23eb7d8446bef71246d853b11edafa93c6ef95 Author: Valay Dave <[email protected]> Date: Sun Mar 20 01:44:57 2022 +0000 Added support for Parameters. - Supporting int, str, bool, float, JSONType commit c9378e9b284657357ad2997f2b492bc2f4aaefac Author: Valay Dave <[email protected]> Date: Sun Mar 20 00:14:10 2022 +0000 Removed todos + added some validation logic. commit 7250a44e1dea1da3464f6f71d0c5188bd314275a Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:45:15 2022 +0000 Fixing logs related change from master. commit d125978619ab666dcf96db330acdca40f41b7114 Merge: 8cdac53 7e210a2 Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:42:48 2022 +0000 Merge branch 'master' into aft-mm commit 8cdac53dd32648455e36955badb8e0ef7b95a2b3 Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:36:47 2022 +0000 making changes sync with master commit 5a93d9f5198c360b2a84ab13a86496986850953c Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:29:47 2022 +0000 Fixed bug when using catch + retry commit 62bc8dff68a6171b3b4222075a8e8ac109f65b4c Author: Valay Dave <[email protected]> Date: Sat Mar 19 22:58:37 2022 +0000 Changed retry setting. commit 563a20036a2dfcc48101f680f29d4917d53aa247 Author: Valay Dave <[email protected]> Date: Sat Mar 19 22:42:57 2022 +0000 Fixed setting `task_id` : - switch task-id from airflow job is to hash to "runid/stepname" - refactor xcom setting variables - added comments commit e2a1e502221dc603385263c82e2c068b9f055188 Author: Valay Dave <[email protected]> Date: Sat Mar 19 17:51:59 2022 +0000 setting retry logic. commit a697b56052210c8f009b68772c902bbf77713202 Author: Valay Dave <[email protected]> Date: Thu Mar 17 01:02:11 2022 +0000 Nit fix. commit 68f13beb17c7e73c0dddc142ef2418675a506439 Author: Valay Dave <[email protected]> Date: Wed Mar 16 20:46:19 2022 +0000 Added @schedule support + readme commit 57bdde54f9ad2c8fe5513dbdb9fd02394664e234 Author: Valay Dave <[email protected]> Date: Tue Mar 15 19:47:06 2022 +0000 Fixed setting run-id / task-id to labels in k8s - Fixed setting run-id has from cli macro - added hashing macro to ensure that jinja template set the correct run-id to k8s labels - commit 3d6c31917297d0be5f9915b13680fc415ddb4421 Author: Valay Dave <[email protected]> Date: Tue Mar 15 05:39:04 2022 +0000 Got linear workflows working on airflow. - Still not feature complete as lots of args are still unfilled / lots of unknows - minor tweek in eks to ensure airflow is k8s compatible. - passing state around via xcom-push - HACK : AWS keys are passed in a shady way. : Reverse this soon. commit db074b8012f76d9d85225a4ceddb2cde8fefa0f4 Author: Valay Dave <[email protected]> Date: Fri Mar 11 12:34:33 2022 -0800 Tweeks commit a9f0468c4721a2017f1b26eb8edcdd80aaa57203 Author: Valay Dave <[email protected]> Date: Tue Mar 1 17:14:47 2022 -0800 some changes based on savin's comments. - Added changes to task datastore for different reason : (todo) Decouple these - Added comments to SFN for reference. - Airflow DAG is no longer dependent on metaflow commit f32d089cd3865927bc7510f24ba3418d859410b6 Author: Valay Dave <[email protected]> Date: Wed Feb 23 00:54:17 2022 -0800 First version of dynamic dag compiler. - Not completely finished code - Creates generic .py file a JSON that is parsed to create Airflow DAG. - Currently only boiler plate to make a linear dag but doesn't execute anything. - Unfinished code. commit d2def665a86d6a6622d6076882c1c2d54044e773 Author: Valay Dave <[email protected]> Date: Sat Feb 19 14:01:47 2022 -0800 more tweeks. commit b176311f166788cc3dfc93354a0c5045a4e6a3d4 Author: Valay Dave <[email protected]> Date: Thu Feb 17 09:04:29 2022 -0800 commit 0 - unfinished code. * Making version compatibility changes. - Minimum support version to 2.2.0 * Task-id macro related logic refactor - Done for better version support. * bug fix: param related task-id setting. * applied black * Reverting `decorators.py` to master * Move sys.path insert earlier in s3op.py (#1098) * Update setup.py (#1099) * Fix for env_escape bug when importing local packages (#1100) * Bump to 2.7.5 (#1102) * Fix another issue with the escape hatch and paths (#1105) * Bump to 2.7.6 (#1106) * add a flag to overwrite config when running metaflow configure sandbox (#1103) * card dev docs tiny fix. (#1108) * Fix an issue with get_cards not respecting a Task's ds-root (#1111) get_cards would not always respect a Task's ds-root leading to cases where a Task has cards but they cannot be accessed because of an invalid path. * Adding support for Azure Blob Storage as a datastore (#1091) * Azure Storage * fix BrokenProcessPool import for older pythons * Fix batch * add azure configure, cannot validate storage account url after all (catch-22 in configure) * rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT * fix indent * clean up kubernetes config cli, add secrets * remove signature * fix fractional resoure handling for batch (#1089) * bump version (#1093) * Fix docstrings for the API reference (no functional changes!) (#1076) Make the docstring format in public APIs compliant with the new API reference framework * Fix issue with get_pinned_conda_libs and metaflow extensions * clean up includefile a bit, remove from_env (not used) * Move a sys.path modification in s3op to __main__ (#1095) In its current location, this could modify the sys.path of the current running metaflow which could have nefarious consequences with the escape hatch which uses sys.path to determine the outside environment's python path. The following scenario would cause issues: - metaflow is installed in the usual path on the system - a conda environment was manually bootstrapped from a directory A. + at this point, sys.path starts with `A` and then contains the other system includes + s3op.py is imported at some point by the Conda installer when it calls `get_many` + this modifies sys.path to insert, at the beginning, the parent of Metaflow; so in this case, sys.path looks something like ['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...] + when the escape hatch trampolines are created, this sys.path is used to determine what the sys.path for the outside interpreter is. + in A, we create: * INFO * metaflow * metaflow_extensions which properly describe the installation of metaflow - when the escape hatch client runs, it runs in the conda environment and uses metaflow created in A. - when the client wants to start the server, this is where we run into issues because, at this point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages' in which it will find metaflow. It will therefore use that metaflow (which is the same as the one linked in A) to start the server. This runs into issues though because A is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`. This will cause issues if multiple extensions are installed there (it will complain about duplicate configurations for example. The `INFO` file typically used to solve this problem is not read as it was not present for the TL metaflow. This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting sys.path when the module is simply included (and not called as a script). * Airflow Support (#1094) * Airflow on Kubernetes minus Foreachs. - Support for all metaflow construct without foreach and sensors Squashed commit of the following: commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1 Author: Valay Dave <[email protected]> Date: Fri Jul 29 01:06:26 2022 +0000 Removed sernsors and banned foreach's commit 8d517c4fecc6568777ad03eca81aaacfa3e91156 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:59:01 2022 +0000 commiting k8s related file from master. commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:54:45 2022 +0000 Uncommented code for foreach support with k8s KubernetesPodOperator version 4.2.0 renamed `resources` to `container_resources` - Check : (https://github.com/apache/airflow/pull/24673) / - (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85) This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator` and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. If it is > 4.2.0 then we set the argument as `container_resources` The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future. So we will only use it for `KuberentesPodOperator` version < 4.2.0 The `resources` argument will also not work for foreach's. commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:31:58 2022 +0000 nit fixes : - fixing comments. - refactor some variable/function names. commit 2079293fbba0d3d862476a7d67b36af8a3389342 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:14:53 2022 +0000 change `token` to `production_token` commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:11:56 2022 +0000 Refactored import Airflow Sensors. commit b1472d5f7a629024ca45e8b83700400d02a4d455 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:08:41 2022 +0000 new comment on `startup_timeout_seconds` env var. commit 6d81b758e8f06911258d26f790029f557488a0d7 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:06:09 2022 +0000 Removing traces of `@airflow_schedule_interval` commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946 Author: Valay Dave <[email protected]> Date: Thu Jul 14 12:43:08 2022 -0700 Foreach polish (valayDave/metaflow#62) * Removing unused imports * Added validation logic for airflow version numbers with foreaches * Removed `airflow_schedule_interval` decorator. * Added production/deployment token related changes - Uses s3 as a backend to store the production token - Token used for avoiding nameclashes - token stored via `FlowDatastore` * Graph type validation for airflow foreachs - Airflow foreachs only support single node fanout. - validation invalidates graphs with nested foreachs * Added configuration about startup_timeout. * Added final todo on `resources` argument of k8sOp - added a commented code block - it needs to be uncommented when airflow releasese the patch for the op - Code seems feature complete keeping aside airflow patch commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9 Author: Valay Dave <[email protected]> Date: Thu Jul 7 19:33:07 2022 +0000 Removed retries from user-defaults. commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43 Author: Valay Dave <[email protected]> Date: Wed Jul 6 16:29:33 2022 +0000 updated pod startup time commit fce2bd263f368dbb78a34ac71f64e13c89277222 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:11 2022 +0000 Adding default 1 retry for any airflow worker. commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321 Author: Valay Dave <[email protected]> Date: Mon Jun 27 01:22:42 2022 +0000 Airflow Foreach Integration - Simple one node foreach-join support as gaurenteed by airflow - Fixed env variable setting issue - introduced MetaflowKuberentesOperator - Created a new operator to allow smootness in plumbing xcom values - Some todos commit d319fa915c558d82f1d127736ce34d3ae0da521d Author: Valay Dave <[email protected]> Date: Fri Jun 24 21:12:09 2022 +0000 simplifying run-id macro. commit 0ffc813b1c4e6ba0103be51520f42d191371741a Author: Valay Dave <[email protected]> Date: Fri Jun 24 11:51:42 2022 -0700 Refactored parameter macro settings. (valayDave/metaflow#60) commit a3a495077f34183d706c0edbe56d6213766bf5f6 Author: Valay Dave <[email protected]> Date: Fri Jun 24 02:05:57 2022 +0000 added comment on need for `start_date` commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec Author: Valay Dave <[email protected]> Date: Tue Jun 21 06:03:56 2022 +0000 Refactored an `id_creator` method. commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:52:05 2022 +0000 refactor : -`RUN_ID_LEN` to `RUN_HASH_ID_LEN` - `TASK_ID_LEN` to `TASK_ID_HASH_LEN` commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2 Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:48:55 2022 +0000 refactored an error string commit 11458188b6c59d044fca0dd2d1f5024ec84f6488 Author: Valay Dave <[email protected]> Date: Mon Jun 20 22:42:36 2022 -0700 addressing savins comments. (#59) - Added many adhoc changes based for some comments. - Integrated secrets and `KUBERNETES_SECRETS` - cleaned up parameter setting - cleaned up setting of scheduling interval - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID` - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate` - Checking if dagfile and flow file are same. - fixing variable names. - checking out `kubernetes_decorator.py` from master (6441ed5) - bug fixing secret setting in airflow. - simplified parameter type parsing logic - refactoring airflow argument parsing code. commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646 Author: Valay Dave <[email protected]> Date: Mon Jun 13 14:02:57 2022 -0700 Addressing Final comments. (#57) - Added dag-run timeout. - airflow related scheduling checks in decorator. - Auto naming sensors if no name is provided - Annotations to k8s operators - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`) - annotation bug fix - setting`workflow-timeout` for only scheduled dags commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:50:49 2022 +0000 k8s bug fix commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:39:50 2022 +0000 removed un-used function commit 70e285e9a7cfbec71fc293508a62c96f33562a01 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:38:37 2022 +0000 Removed unused `sanitize_label` function commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:37:34 2022 +0000 GPU support added + container naming same as argo commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:25:17 2022 +0000 Refactored sensors to different files + bug fix - bug caused due `util.compress_list`. - The function doesn't play nice with strings with variety of characters. - Ensured that exceptions are handled appropriately. - Made new file for each sensor under `airflow.sensors` module. commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673 Author: Valay Dave <[email protected]> Date: Sat Jun 4 01:41:49 2022 +0000 ran black. commit 558c82f65b383ed0d61ded6bc80326471e284550 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:32:48 2022 -0700 Moving information from airflow_utils to compiler (#56) - commenting todos to organize unfinished changes. - some environment variables set via`V1EnvVar` - `client.V1ObjectFieldSelector` mapped env vars were not working in json form - Moving k8s operator import into its own function. - env vars moved. commit 9bb5f638792a671164ec95891e97f599e9a3385f Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:06:03 2022 +0000 added mising Run-id prefixes to variables. - merged `hash` and `dash_connect` filters. commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:00:22 2022 +0000 nit fix : variable name change. commit 660756f952ebd92ba1e26d7f908b81036c31ff10 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:58:34 2022 +0000 nit fixes to dag.py's templating variables. commit 1202f5bc92f76df52b5957f11c8574cadfa62196 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:56:53 2022 +0000 Fixed defaults passing - Addressed comments for airflow.py commit b9387dd428c1a37f9a3bfe2c72cab475da708c02 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:52:24 2022 +0000 Following Changes: - Refactors setting scheduling interval - refactor dag file creating function - refactored is_active to is_paused_upon_creation - removed catchup commit 054e3f389febc6c447494a1dedb01228f5f5650f Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:33:25 2022 +0000 Multiple Changes based on comments: 1. refactored `create_k8s_args` into _to_job 2. Addressed comments for snake casing 3. refactored `attrs` for simplicity. 4. refactored `metaflow_parameters` to `parameters`. 5. Refactored setting of `input_paths` commit d481b2fca7914b6b657a69af407cfe1a894a46dc Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:42:24 2022 +0000 Removed Sensor metadata extraction. commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3 Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:30:34 2022 +0000 porting savin's comments - next changes : addressing comments. commit 3f2353a647e53bc240e28792769c42a71ea8f8c9 Merge: d370ffb c1ff469 Author: Valay Dave <[email protected]> Date: Thu Jul 28 23:52:16 2022 +0000 Merge branch 'master' into airflow commit d370ffb248411ad4675f9d55de709dbd75d3806e Merge: a82f144 e4eb751 Author: Valay Dave <[email protected]> Date: Thu Jul 14 19:38:48 2022 +0000 Merge branch 'master' into airflow commit a82f1447b414171fc5611758cb6c12fc692f55f9 Merge: bdb1f0d 6f097e3 Author: Valay Dave <[email protected]> Date: Wed Jul 13 00:35:49 2022 +0000 Merge branch 'master' into airflow commit bdb1f0dd248d01318d4a493c75b6f54248c7be64 Merge: 8511215 f9a4968 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:51 2022 +0000 Merge branch 'master' into airflow commit 85112158cd352cb7de95a2262c011c6f43d98283 Author: Valay Dave <[email protected]> Date: Tue Jun 21 02:53:11 2022 +0000 Bug fix from master merge. commit 90c06f12bb14eda51c6a641766c5f67d6763abaa Merge: 0fb73af 6441ed5 Author: Valay Dave <[email protected]> Date: Mon Jun 20 21:20:20 2022 +0000 Merge branch 'master' into airflow commit 0fb73af8af9fca2875261e3bdd305a0daab1b229 Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:53:10 2022 +0000 squashing bugs after changes from master. commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d Merge: 7bdf662 ffff49b Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:20:38 2022 +0000 Merge branch 'master' into af-mmr commit 7bdf662e14966b929b8369c65d5bd3bbe5741937 Author: Valay Dave <[email protected]> Date: Mon May 16 17:42:38 2022 -0700 Airflow sensor api (#3) * Fixed run-id setting - Change gaurentees that multiple dags triggered at same moment have unique run-id * added allow multiple in `Decorator` class * Airflow sensor integration. >> support added for : - ExternalTaskSensor - S3KeySensor - SqlSensor >> sensors allow multiple decorators >> sensors accept those arguments which are supported by airflow * Added `@airflow_schedule_interval` decorator * Fixing bug run-id related in env variable setting. commit 2604a29452e794354cf4c612f48bae7cf45856ee Author: Valay Dave <[email protected]> Date: Thu Apr 21 18:26:59 2022 +0000 Addressed comments. commit 584e88b679fed7d6eec8ce564bf3707359170568 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:33:55 2022 +0000 fixed printing bug commit 169ac1535e5567149d94749ddaf70264e882d62c Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:30:59 2022 +0000 Option help bug fix. commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:25:54 2022 +0000 variable renamemetaflow_specific_args commit 0c779abcd1d9574878da6de8183461b53e0da366 Merge: d299b13 5a61508 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:23:10 2022 +0000 Merge branch 'airflow-tests' into airflow commit 5a61508e61583b567ef8d3fea04e049d74a6d973 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:22:54 2022 +0000 Removing un-used code / resolved-todos. commit d030830f2543f489a1c4ebd17da1b47942f041d6 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:06:03 2022 +0000 ran black, commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45 Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:04:19 2022 +0000 Merge branch 'master' into airflow-tests commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:02:37 2022 +0000 Merge branch 'master' into airflow commit f2cb3197725f11520da0d49cbeef8de215c243eb Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:54:03 2022 +0000 reverting change. commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4 Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:47:41 2022 +0000 3 changes: - Removing s3 dep - remove uesless import - added `deployed_on` in dag file template commit c6afba95f5ec05acf7f33fd3228cffd784556e3b Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:50:52 2022 +0000 Fixed passing secrets with kubernetes. commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:04:22 2022 +0000 Refactored code . - removed compute/k8s.py - Moved k8s code to airflow_compiler.py - ran isort to airflow_compiler.py commit d1c343dbbffbddbebd2aeda26d6846e595144e0b Author: Valay Dave <[email protected]> Date: Fri Apr 15 18:02:25 2022 +0000 Added validations about: - un-supported decorators - foreach Changed where validations are done to not save the package. commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce Author: Valay Dave <[email protected]> Date: Fri Apr 15 03:34:26 2022 +0000 Fixing mf log related bug - No double logging on metaflow. commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:10:51 2022 +0000 Removed usless code WRT project decorator. commit 5ad9a3949e351b0ac13f11df13446953932e8ffc Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:03:19 2022 +0000 Remove readme. commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:02:38 2022 +0000 Made file path required arguement. commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:01:07 2022 +0000 changed `--is-active`->`--is-paused-upon-creation` - dags are active by default. commit 5b98f937a62ee74de8aed8b0efde5045a28f068b Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:55:46 2022 +0000 shortened length of run-id and task-id hashes. commit e53426eaa4b156e8bd70ae7510c2e7c66745d101 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:41:32 2022 +0000 …
npow
added a commit
to npow/metaflow
that referenced
this pull request
Jan 12, 2023
* Don't explicitly break py2 support (#962) * Don't explicitly break py2 support * Typo * Typo * Pass the paths used by the external interpreter to the one launching the Env Escape server (#960) We were currently not reflecting possible values passed using PYTHONPATH or programmatically added to the interpreter launching the environment escape server. * Bump follow-redirects in /metaflow/plugins/cards/ui (#966) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.7...v1.14.9) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Adding export of graph to JSON for cli (#955) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS (#972) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS * Ran black * emit app tag for AWS Batch jobs (#970) * Bump to 2.5.3 (#974) * Extension packaging improvements (#959) * Fixed dual distribution import in some cases; fixed get_plugin_cli extension If sys.path contains the same directory multiple times, metadata.distributions() will list a package multiple times which breaks the import mechanism we have. This is addressed by skipping duplicate packages. get_plugin_cli for extensions wasn't evaluated late enough therefore not allowing additional parameters to be added. * Fix issue with metaflow_extensions and Conda environment In a Conda environment, it is not possible to re-resolve all the metaflow_extension packages since they appear as a single directory. In that case, we resolve on the INFO file to give us the proper information. Also fixed the way metaflow_extensions are added to the Conda environment to avoid leaking more information than needed. * Validate configuration files for extensions and de-duplicate for internal ns packages * Update packaging of metaflow_extensions packages Packaging is now handled directly by extension_support.py and each distribution/package can define its own suffixes to include. * Rework the import mechanism for modules contained in metaflow_extensions This change does away with the deprecated load_module and also improves handling of loading children modules and _orig modules * Address comments * Squash merge of origin/master * Fix issue when packages were left out * Merge get_pinned_conda_libs * Fix issue with __init__.py in non-distribution packages * Moving card import changes on top of Romain's branch (#973) * Moving card import changes on top of Romain's branch - tiny bug fix to extensions. - Added card related refactor to support mfextinit and regular package. - removing card packaging related code in decorator + everywhere else. * Tiny bug fix on top of romain's new changes. * removing unneccessary logic * Added a bit more inline documentation and addressed documentation comments * Properly handle parsing of package requirements * Fix Pylint errors in some case of metaflow_extensions module aliasing * Properly handle case of files at the root of PYTHONPATH package * Tests for Extensions (#978) * Added tests for extensions. Checking if they work. * dummy commit to see if things work. * fix * debug * bug fix. * possible fix. * tweeking context * Bug fix to tests. * dummy commit * bug fix * Added extension test to core tests. - remove seperate gh action * removing files. * fix * added extension test to py3 context - remove redundant complexity. Co-authored-by: Valay Dave <[email protected]> * pass DEFAULT_AWS_CLIENT_PROVIDER to remote tasks (#982) * Fixing some hacky plumbing in card test suite (#967) * Fixing some hacky plumbing for card tests. - Added `--file` arg to card list cli - Using files to get the information written by cli - changes to test to use files instead of stdout - Piping stderrors to stderrr to capture card not found errors. * Removing `\n` from all card tests assertions - `\n` was there because earlier we read stdout. - Now since we read files, it will not be needed. * Simplify mflog (#979) * Fix extension root determination in some cases + card tweaks (#989) * Use importlib_metadata 2.8.3 for Python 3.4, 3.5 but 4.8.3 for Python 3.6+ (#988) * Use importlib_metadata 2.8.3 for Python 3.4 and 3.5 but 4.8.3 for Python 3.6+ This is required because in importlib_metadata 3.4.0, a field called `_normalized_name` was introduced and is relied on by later versions of importlib_metadata. When importlib_metadata looks for distributions, it queries all registered/installed importlib_metadata and if an older version (like 2.8.3) returns something that does not have this field, it causes a crash. * Add __init__.py files to make the vendored package non namespace * Allow configuring the root directory where artifacts go when pulling from S3. (#991) Co-authored-by: kgullikson <[email protected]> * Add an option to pass a `role` argument to S3() to provide a role (#987) * Add an option to pass a `role` argument to S3() to provide a role for the S3 client * Move to partial to be able to pickle ops * Address comments * bump version to 2.5.4 (#993) * Bug fix for datetime index in default card (#981) * Bug fix for datetime_indexes. * ran black * Bump minimist from 1.2.5 to 1.2.6 in /metaflow/plugins/cards/ui (#995) Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6. - [Release notes](https://github.com/substack/minimist/releases) - [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6) --- updated-dependencies: - dependency-name: minimist dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix black - set upper bound for click version (#1006) * Add tags to `current` singleton (#1019) * first attempt * fix _set_env param typo * add assertions for tags to current singleton test * Dispatch Metaflow flows to Argo Workflows (#992) * Dispatch Metaflow flows to Argo Workflows * Remove spurious files * Make black happy * Make black happy again * Remove spurious TODOs * Handle non-numeric task ids * Fix typos * don't actively automount service account tokens * make black happy * Update @kubernetes * remove spurious commits * fix issue with hyphens * raise memory for tests * change limits to requests * pin back test resources * update refresh timeout * stylistic nits * Add lint check * make black happy * support gpus * drop extra bracket * fix * Skip sleep if not retryting S3 operations (#1001) * Don't sleep if not retrying S3 operations * Remove unused env variable * VSCode is too smart * Bump minor version for release (#1022) * Alternate way of adding metadata to cloned tasks (#1003) * Alternate way of adding metadata to cloned tasks This adds it on the client side * Clean up metadata passing. Also optimize use within the client * Typo * Fix black test * Keep black happy * Added tests * Ignore namespace when getting origin task * Bump cross-fetch from 3.1.4 to 3.1.5 in /metaflow/plugins/cards/ui (#1027) Bumps [cross-fetch](https://github.com/lquixada/cross-fetch) from 3.1.4 to 3.1.5. - [Release notes](https://github.com/lquixada/cross-fetch/releases) - [Commits](https://github.com/lquixada/cross-fetch/compare/v3.1.4...v3.1.5) --- updated-dependencies: - dependency-name: cross-fetch dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed bug from #1023 (#1025) * current.pathspec to return None when used outside Flow (#1033) * current.pathspec to return None when used outside Flow * formatted with black * Fixing bug in the `card list` command (#1044) - echo_always defaulted to stderr :( * Enable MinIO/EMC-ECS as blob stores for Metaflow (#1045) * Enable EMCECS * Enable MinIO & EMC-ECS blob stores for Metaflow * fix InvalidRange issue * [tag-mutation-project] Stash system info in task metadata (#1039) * stash system info as metadata (in addition to system_tags today) * improve * avoid parsing on ":" for user id * remove date - it's not well defined and created_at is available * python_version is in metadata now, and can vary between orig and resume runs * address CR comments * add the comments * Address issues with S3 get and ranges (#1034) * Address issues with S3 get and ranges This addresses two issues: - get and get_many would not properly populate the range_info in S3Object - get_many on different ranges of the same file would return incorrect results * Add one more test (merge with Netflix internal tests) * Put Run() test behind a check * Address comments * Fix _new_task calling bug in LocalMetadataProvider (#1046) * Release 2.6.1 (#1047) * Support default secrets (#1048) * Run id file should be written prior to execution when resuming (#1051) * Run id file should be written prior to execution when resuming * separate run id test so we can exclude from batch * release 2.6.2 (#1052) * Fix instance metadata calls for IMDSV2 (#1053) * bump version to 2.6.3 (#1054) * Small reorder of imports to make things more consistent (#1055) * [tag-mutation-project] Local metadata provider - object read paths to return ancestral Run tags (#1043) * wip * Fix nits * typo fix. (#1068) * This is the Jackie's tag changes with an additional test and a fix. (#1063) * rebase on master * add replace_many as candidate subcommand * updates from some CR suggestions * propagate opt rename * more changes follow CR * improve consistency guarantees of local metadata tag mutations (retries if a race was lost) * write comment * use replace-many logic to back "tag replace" * finalize client API for tag operations * fix retry logic * Add task.tags == run.tags check in test * [tag-mutation-project] Add tag mutation support to ServiceMetadataProvider * fix version * Enable TagMutationTest in batch and k8s * Bogus commit * minor UX fixes on CLI * Fully support CliCheck for TagMutationTest * add tag and tag set validation to local metadata code path Validate tags on run / resume paths too * flat nor flatten on CLI * Logs really do come via stdout - related to CliCheck refactor * argo and step functions - validate tags as soon as possible (before other user messaging) * no capture_output pre 3.7 * Add another test for tag mutation * Properly pass down the metadata provider There were two cases when the metadata provider was not properly passed down to the client: - when calling the client within a flow - when invoking the `tag` command This fixes both issues and also allows the user to introspect the metadata provider and datastore being used for the flow * Do not make public metadata and datastore in current object * Typo Co-authored-by: Jackie Tung <[email protected]> Co-authored-by: jackie-ob <[email protected]> * Add two options to resume: the ability to specify a run-id and the ab… (#1059) * Add two options to resume: the ability to specify a run-id and the ability to only clone tasks The scenario for this use is mainly for external schedulers that are capable of "resuming" failed runs by only re-executing the ones that failed or didn't execute the first time. We need a way for Metaflow to "clone" the tasks that are not re-executed. * Small formatting fix * Made clone-only resume possibly re-entrant * Addressed comments * Clarify the reentrant behavior of resume * Improve sidecar message handling (#1057) * Improve sidecar message handling Several changes: - allow side-passing of a static "context" to the sidecar process to keep messages as small as possible - clean up the interfaces and make it clearer what is what (ie: there is an emitter inside the process and then a sidecar that runs outside) - better error checking - better handling of shutdown (sidecars will now properly get a shutdown message giving them the opportunity to clean up -- this may add a bit of latency at the end of the program execution. Also improved message when plugins don't load in case of an issue with Metaflow extensions * Add the possibility to pass additional context to sidecars This allows more information to be provided after the sidecar is created but still maintains small message sizes * Better handling of initial context message for sidecars In the previous implementation, a failure to send the initial context message could crash the flow. This change properly handles that message like any other message (so has no impact on the flow itself). It also implements better retry policies to send the context. Finally, this change also improves error handling: if a message fails to send, in the previous implementation, it was likely that the next successful one would be an invalid json. * Improve handling of corner conditions on message send * Restart sidecar faster when sidecar process dies * Properly send context after sidecar restart * Remove the specific context from monitors/loggers Subclasses can now handle contexts directly in the subclass. Refactoring also cleaned up what a sidecar and a sidecar worker do by refactoring some of the code out. * Typo: fix names * Minor typos for sidecar changes * Clean up sidecars * More cleanups for sidecars * Addressed final comments for sidecars * Small typo in NullEventLogger.send (#1071) Pushing directly per our conversation with Savin * Bump version to 2.7.0 * Change behavior of env escape when module is present (#1072) When a module configured in the env escape was present in the local environment, we would raise an error. We now print a message and use the module present in the environment (not the environment escape). This allows the env escape modules to provide a "fallback" in case the module is not present but to still use the local module if present * Bump version to 2.7.1 (#1073) * Fix an issue with the environment escape server directory (#1074) The environment escape server would launch in the cwd. This could cause issues if, for example, a `metaflow` directory existed in that cwd (because it would then take precedence over the actual metaflow. To remediate this situation, the server is launched in the configuration directory (where we know what there is). Also took care of a quick annoyance of having to create an empty overrides.py file for everything. This is now no longer required * Support session_vars and client_params in S3() (#1069) * Support session_vars and client_params in S3() This allows the user to pass down session variables and client parameters to set when getting a S3 client. * Addressed comments about default configuration * Support M1 Macs (#1077) * changes to @kubernetes for sandbox (#1079) * Changed error to warning in AWS retry (#1081) Co-authored-by: Preetam Joshi <[email protected]> * bump to 2.7.2 (#1083) * Fixed s3util test (#1084) * Fixing s3util test caset * added comment Co-authored-by: Preetam Joshi <[email protected]> * Bump svelte from 3.46.1 to 3.49.0 in /metaflow/plugins/cards/ui (#1086) Bumps [svelte](https://github.com/sveltejs/svelte) from 3.46.1 to 3.49.0. - [Release notes](https://github.com/sveltejs/svelte/releases) - [Changelog](https://github.com/sveltejs/svelte/blob/master/CHANGELOG.md) - [Commits](https://github.com/sveltejs/svelte/compare/v3.46.1...v3.49.0) --- updated-dependencies: - dependency-name: svelte dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Metadata version check flag (#1088) * Bump terser from 5.10.0 to 5.14.2 in /metaflow/plugins/cards/ui (#1090) Bumps [terser](https://github.com/terser/terser) from 5.10.0 to 5.14.2. - [Release notes](https://github.com/terser/terser/releases) - [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md) - [Commits](https://github.com/terser/terser/commits) --- updated-dependencies: - dependency-name: terser dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix r CI deps (#1092) * fix r CI deps * fix fractional resoure handling for batch (#1089) * bump version (#1093) * Fix docstrings for the API reference (no functional changes!) (#1076) Make the docstring format in public APIs compliant with the new API reference framework * Move a sys.path modification in s3op to __main__ (#1095) In its current location, this could modify the sys.path of the current running metaflow which could have nefarious consequences with the escape hatch which uses sys.path to determine the outside environment's python path. The following scenario would cause issues: - metaflow is installed in the usual path on the system - a conda environment was manually bootstrapped from a directory A. + at this point, sys.path starts with `A` and then contains the other system includes + s3op.py is imported at some point by the Conda installer when it calls `get_many` + this modifies sys.path to insert, at the beginning, the parent of Metaflow; so in this case, sys.path looks something like ['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...] + when the escape hatch trampolines are created, this sys.path is used to determine what the sys.path for the outside interpreter is. + in A, we create: * INFO * metaflow * metaflow_extensions which properly describe the installation of metaflow - when the escape hatch client runs, it runs in the conda environment and uses metaflow created in A. - when the client wants to start the server, this is where we run into issues because, at this point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages' in which it will find metaflow. It will therefore use that metaflow (which is the same as the one linked in A) to start the server. This runs into issues though because A is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`. This will cause issues if multiple extensions are installed there (it will complain about duplicate configurations for example. The `INFO` file typically used to solve this problem is not read as it was not present for the TL metaflow. This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting sys.path when the module is simply included (and not called as a script). * Airflow Support (#1094) * Airflow on Kubernetes minus Foreachs. - Support for all metaflow construct without foreach and sensors Squashed commit of the following: commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1 Author: Valay Dave <[email protected]> Date: Fri Jul 29 01:06:26 2022 +0000 Removed sernsors and banned foreach's commit 8d517c4fecc6568777ad03eca81aaacfa3e91156 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:59:01 2022 +0000 commiting k8s related file from master. commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:54:45 2022 +0000 Uncommented code for foreach support with k8s KubernetesPodOperator version 4.2.0 renamed `resources` to `container_resources` - Check : (https://github.com/apache/airflow/pull/24673) / - (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85) This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator` and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. If it is > 4.2.0 then we set the argument as `container_resources` The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future. So we will only use it for `KuberentesPodOperator` version < 4.2.0 The `resources` argument will also not work for foreach's. commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:31:58 2022 +0000 nit fixes : - fixing comments. - refactor some variable/function names. commit 2079293fbba0d3d862476a7d67b36af8a3389342 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:14:53 2022 +0000 change `token` to `production_token` commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:11:56 2022 +0000 Refactored import Airflow Sensors. commit b1472d5f7a629024ca45e8b83700400d02a4d455 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:08:41 2022 +0000 new comment on `startup_timeout_seconds` env var. commit 6d81b758e8f06911258d26f790029f557488a0d7 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:06:09 2022 +0000 Removing traces of `@airflow_schedule_interval` commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946 Author: Valay Dave <[email protected]> Date: Thu Jul 14 12:43:08 2022 -0700 Foreach polish (valayDave/metaflow#62) * Removing unused imports * Added validation logic for airflow version numbers with foreaches * Removed `airflow_schedule_interval` decorator. * Added production/deployment token related changes - Uses s3 as a backend to store the production token - Token used for avoiding nameclashes - token stored via `FlowDatastore` * Graph type validation for airflow foreachs - Airflow foreachs only support single node fanout. - validation invalidates graphs with nested foreachs * Added configuration about startup_timeout. * Added final todo on `resources` argument of k8sOp - added a commented code block - it needs to be uncommented when airflow releasese the patch for the op - Code seems feature complete keeping aside airflow patch commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9 Author: Valay Dave <[email protected]> Date: Thu Jul 7 19:33:07 2022 +0000 Removed retries from user-defaults. commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43 Author: Valay Dave <[email protected]> Date: Wed Jul 6 16:29:33 2022 +0000 updated pod startup time commit fce2bd263f368dbb78a34ac71f64e13c89277222 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:11 2022 +0000 Adding default 1 retry for any airflow worker. commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321 Author: Valay Dave <[email protected]> Date: Mon Jun 27 01:22:42 2022 +0000 Airflow Foreach Integration - Simple one node foreach-join support as gaurenteed by airflow - Fixed env variable setting issue - introduced MetaflowKuberentesOperator - Created a new operator to allow smootness in plumbing xcom values - Some todos commit d319fa915c558d82f1d127736ce34d3ae0da521d Author: Valay Dave <[email protected]> Date: Fri Jun 24 21:12:09 2022 +0000 simplifying run-id macro. commit 0ffc813b1c4e6ba0103be51520f42d191371741a Author: Valay Dave <[email protected]> Date: Fri Jun 24 11:51:42 2022 -0700 Refactored parameter macro settings. (valayDave/metaflow#60) commit a3a495077f34183d706c0edbe56d6213766bf5f6 Author: Valay Dave <[email protected]> Date: Fri Jun 24 02:05:57 2022 +0000 added comment on need for `start_date` commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec Author: Valay Dave <[email protected]> Date: Tue Jun 21 06:03:56 2022 +0000 Refactored an `id_creator` method. commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:52:05 2022 +0000 refactor : -`RUN_ID_LEN` to `RUN_HASH_ID_LEN` - `TASK_ID_LEN` to `TASK_ID_HASH_LEN` commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2 Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:48:55 2022 +0000 refactored an error string commit 11458188b6c59d044fca0dd2d1f5024ec84f6488 Author: Valay Dave <[email protected]> Date: Mon Jun 20 22:42:36 2022 -0700 addressing savins comments. (#59) - Added many adhoc changes based for some comments. - Integrated secrets and `KUBERNETES_SECRETS` - cleaned up parameter setting - cleaned up setting of scheduling interval - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID` - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate` - Checking if dagfile and flow file are same. - fixing variable names. - checking out `kubernetes_decorator.py` from master (6441ed5) - bug fixing secret setting in airflow. - simplified parameter type parsing logic - refactoring airflow argument parsing code. commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646 Author: Valay Dave <[email protected]> Date: Mon Jun 13 14:02:57 2022 -0700 Addressing Final comments. (#57) - Added dag-run timeout. - airflow related scheduling checks in decorator. - Auto naming sensors if no name is provided - Annotations to k8s operators - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`) - annotation bug fix - setting`workflow-timeout` for only scheduled dags commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:50:49 2022 +0000 k8s bug fix commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:39:50 2022 +0000 removed un-used function commit 70e285e9a7cfbec71fc293508a62c96f33562a01 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:38:37 2022 +0000 Removed unused `sanitize_label` function commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:37:34 2022 +0000 GPU support added + container naming same as argo commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:25:17 2022 +0000 Refactored sensors to different files + bug fix - bug caused due `util.compress_list`. - The function doesn't play nice with strings with variety of characters. - Ensured that exceptions are handled appropriately. - Made new file for each sensor under `airflow.sensors` module. commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673 Author: Valay Dave <[email protected]> Date: Sat Jun 4 01:41:49 2022 +0000 ran black. commit 558c82f65b383ed0d61ded6bc80326471e284550 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:32:48 2022 -0700 Moving information from airflow_utils to compiler (#56) - commenting todos to organize unfinished changes. - some environment variables set via`V1EnvVar` - `client.V1ObjectFieldSelector` mapped env vars were not working in json form - Moving k8s operator import into its own function. - env vars moved. commit 9bb5f638792a671164ec95891e97f599e9a3385f Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:06:03 2022 +0000 added mising Run-id prefixes to variables. - merged `hash` and `dash_connect` filters. commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:00:22 2022 +0000 nit fix : variable name change. commit 660756f952ebd92ba1e26d7f908b81036c31ff10 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:58:34 2022 +0000 nit fixes to dag.py's templating variables. commit 1202f5bc92f76df52b5957f11c8574cadfa62196 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:56:53 2022 +0000 Fixed defaults passing - Addressed comments for airflow.py commit b9387dd428c1a37f9a3bfe2c72cab475da708c02 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:52:24 2022 +0000 Following Changes: - Refactors setting scheduling interval - refactor dag file creating function - refactored is_active to is_paused_upon_creation - removed catchup commit 054e3f389febc6c447494a1dedb01228f5f5650f Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:33:25 2022 +0000 Multiple Changes based on comments: 1. refactored `create_k8s_args` into _to_job 2. Addressed comments for snake casing 3. refactored `attrs` for simplicity. 4. refactored `metaflow_parameters` to `parameters`. 5. Refactored setting of `input_paths` commit d481b2fca7914b6b657a69af407cfe1a894a46dc Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:42:24 2022 +0000 Removed Sensor metadata extraction. commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3 Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:30:34 2022 +0000 porting savin's comments - next changes : addressing comments. commit 3f2353a647e53bc240e28792769c42a71ea8f8c9 Merge: d370ffb c1ff469 Author: Valay Dave <[email protected]> Date: Thu Jul 28 23:52:16 2022 +0000 Merge branch 'master' into airflow commit d370ffb248411ad4675f9d55de709dbd75d3806e Merge: a82f144 e4eb751 Author: Valay Dave <[email protected]> Date: Thu Jul 14 19:38:48 2022 +0000 Merge branch 'master' into airflow commit a82f1447b414171fc5611758cb6c12fc692f55f9 Merge: bdb1f0d 6f097e3 Author: Valay Dave <[email protected]> Date: Wed Jul 13 00:35:49 2022 +0000 Merge branch 'master' into airflow commit bdb1f0dd248d01318d4a493c75b6f54248c7be64 Merge: 8511215 f9a4968 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:51 2022 +0000 Merge branch 'master' into airflow commit 85112158cd352cb7de95a2262c011c6f43d98283 Author: Valay Dave <[email protected]> Date: Tue Jun 21 02:53:11 2022 +0000 Bug fix from master merge. commit 90c06f12bb14eda51c6a641766c5f67d6763abaa Merge: 0fb73af 6441ed5 Author: Valay Dave <[email protected]> Date: Mon Jun 20 21:20:20 2022 +0000 Merge branch 'master' into airflow commit 0fb73af8af9fca2875261e3bdd305a0daab1b229 Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:53:10 2022 +0000 squashing bugs after changes from master. commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d Merge: 7bdf662 ffff49b Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:20:38 2022 +0000 Merge branch 'master' into af-mmr commit 7bdf662e14966b929b8369c65d5bd3bbe5741937 Author: Valay Dave <[email protected]> Date: Mon May 16 17:42:38 2022 -0700 Airflow sensor api (#3) * Fixed run-id setting - Change gaurentees that multiple dags triggered at same moment have unique run-id * added allow multiple in `Decorator` class * Airflow sensor integration. >> support added for : - ExternalTaskSensor - S3KeySensor - SqlSensor >> sensors allow multiple decorators >> sensors accept those arguments which are supported by airflow * Added `@airflow_schedule_interval` decorator * Fixing bug run-id related in env variable setting. commit 2604a29452e794354cf4c612f48bae7cf45856ee Author: Valay Dave <[email protected]> Date: Thu Apr 21 18:26:59 2022 +0000 Addressed comments. commit 584e88b679fed7d6eec8ce564bf3707359170568 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:33:55 2022 +0000 fixed printing bug commit 169ac1535e5567149d94749ddaf70264e882d62c Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:30:59 2022 +0000 Option help bug fix. commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:25:54 2022 +0000 variable renamemetaflow_specific_args commit 0c779abcd1d9574878da6de8183461b53e0da366 Merge: d299b13 5a61508 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:23:10 2022 +0000 Merge branch 'airflow-tests' into airflow commit 5a61508e61583b567ef8d3fea04e049d74a6d973 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:22:54 2022 +0000 Removing un-used code / resolved-todos. commit d030830f2543f489a1c4ebd17da1b47942f041d6 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:06:03 2022 +0000 ran black, commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45 Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:04:19 2022 +0000 Merge branch 'master' into airflow-tests commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:02:37 2022 +0000 Merge branch 'master' into airflow commit f2cb3197725f11520da0d49cbeef8de215c243eb Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:54:03 2022 +0000 reverting change. commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4 Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:47:41 2022 +0000 3 changes: - Removing s3 dep - remove uesless import - added `deployed_on` in dag file template commit c6afba95f5ec05acf7f33fd3228cffd784556e3b Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:50:52 2022 +0000 Fixed passing secrets with kubernetes. commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:04:22 2022 +0000 Refactored code . - removed compute/k8s.py - Moved k8s code to airflow_compiler.py - ran isort to airflow_compiler.py commit d1c343dbbffbddbebd2aeda26d6846e595144e0b Author: Valay Dave <[email protected]> Date: Fri Apr 15 18:02:25 2022 +0000 Added validations about: - un-supported decorators - foreach Changed where validations are done to not save the package. commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce Author: Valay Dave <[email protected]> Date: Fri Apr 15 03:34:26 2022 +0000 Fixing mf log related bug - No double logging on metaflow. commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:10:51 2022 +0000 Removed usless code WRT project decorator. commit 5ad9a3949e351b0ac13f11df13446953932e8ffc Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:03:19 2022 +0000 Remove readme. commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:02:38 2022 +0000 Made file path required arguement. commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:01:07 2022 +0000 changed `--is-active`->`--is-paused-upon-creation` - dags are active by default. commit 5b98f937a62ee74de8aed8b0efde5045a28f068b Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:55:46 2022 +0000 shortened length of run-id and task-id hashes. commit e53426eaa4b156e8bd70ae7510c2e7c66745d101 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:41:32 2022 +0000 Removing un-used args. commit 72cbbfc7424f9be415c22d9144b16a0953f15295 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:39:59 2022 +0000 Moved exceptions to airflow compiler commit b2970ddaa86c393c8abb7f203f6507c386ecbe00 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:33:02 2022 +0000 Changes based on PR comments: - removed airflow xcom push file , moved to decorator code - removed prefix configuration - nit fixes. commit 9e622bac5a75eb9e7a6594d8fa0e47f076634b44 Author: Valay Dave <[email protected]> Date: Mon Apr 11 20:39:00 2022 +0000 Removing un-used code paths + code cleanup commit 7425f62cff2c9128eea785223ddeb40fa2d8f503 Author: Valay Dave <[email protected]> Date: Mon Apr 11 19:45:04 2022 +0000 Fixing bug fix in schedule. commit eb775cbadd1d2d2c90f160a95a0f42c8ff0d7f4c Author: Valay Dave <[email protected]> Date: Sun Apr 10 02:52:59 2022 +0000 Bug fixes WRT Kubernetes secrets + k8s deployments. - Fixing some error messages. - Added some comments. commit 04c92b92c312a4789d3c1e156f61ef57b08dba9f Author: Valay Dave <[email protected]> Date: Sun Apr 10 01:20:53 2022 +0000 Added secrets support. commit 4a0a85dff77327640233767e567aee2b379ac13e Author: Valay Dave <[email protected]> Date: Sun Apr 10 00:11:46 2022 +0000 Bug fix. commit af91099c0a30c26b58d58696a3ef697ec49a8503 Author: Valay Dave <[email protected]> Date: Sun Apr 10 00:03:34 2022 +0000 bug fix. commit c17f04a253dfe6118e2779db79da9669aa2fcef2 Author: Valay Dave <[email protected]> Date: Sat Apr 9 23:55:41 2022 +0000 Bug fix in active defaults. commit 0d372361297857076df6af235d1de7005ac1544a Author: Valay Dave <[email protected]> Date: Sat Apr 9 23:54:02 2022 +0000 @project, @schedule, default active dag support. - Added a flag to allow setting dag as active on creation - Airflow compatible schedule interval - Project name fixes. commit 5c97b15cb11b5e8279befc5b14c239463750e9b7 Author: Valay Dave <[email protected]> Date: Thu Apr 7 21:15:18 2022 +0000 Max workers and worker pool support. commit 9c973f2f44c3cb3a98e3e63f6e4dcef898bc8bf2 Author: Valay Dave <[email protected]> Date: Thu Apr 7 19:34:33 2022 +0000 Adding exceptions for missing features. commit 2a946e2f083a34b4b6ed84c70aebf96b084ee8a2 Author: Valay Dave <[email protected]> Date: Mon Mar 28 19:34:11 2022 +0000 2 changes : - removed hacky line - added support to directly throw dags in s3. commit e0772ec1bad473482c6fd19f8c5e8b9845303c0a Author: Valay Dave <[email protected]> Date: Wed Mar 23 22:38:20 2022 +0000 fixing bugs in service account setting commit 874b94aeeabc664f12551864eff9d8fdc24dc37b Author: Valay Dave <[email protected]> Date: Sun Mar 20 23:49:15 2022 +0000 Added support for Branching with Airflow - remove `next` function in `AirflowTask` - `AirflowTask`s has no knowledge of next tasks. - removed todos + added some todos - Graph construction on airflow side using graph_structure datastructure. - graph_structure comes from`FlowGraph.output_steps()[1]` commit 8e9f649bd8c51171c38a1e5af70a44a85e7009ca Author: Valay Dave <[email protected]> Date: Sun Mar 20 02:33:04 2022 +0000 Added hacky line commit fd5db04cf0a81b14efda5eaf40cd9227e2bac0d3 Author: Valay Dave <[email protected]> Date: Sun Mar 20 02:06:38 2022 +0000 Removed hacky line. commit 5b23eb7d8446bef71246d853b11edafa93c6ef95 Author: Valay Dave <[email protected]> Date: Sun Mar 20 01:44:57 2022 +0000 Added support for Parameters. - Supporting int, str, bool, float, JSONType commit c9378e9b284657357ad2997f2b492bc2f4aaefac Author: Valay Dave <[email protected]> Date: Sun Mar 20 00:14:10 2022 +0000 Removed todos + added some validation logic. commit 7250a44e1dea1da3464f6f71d0c5188bd314275a Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:45:15 2022 +0000 Fixing logs related change from master. commit d125978619ab666dcf96db330acdca40f41b7114 Merge: 8cdac53 7e210a2 Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:42:48 2022 +0000 Merge branch 'master' into aft-mm commit 8cdac53dd32648455e36955badb8e0ef7b95a2b3 Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:36:47 2022 +0000 making changes sync with master commit 5a93d9f5198c360b2a84ab13a86496986850953c Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:29:47 2022 +0000 Fixed bug when using catch + retry commit 62bc8dff68a6171b3b4222075a8e8ac109f65b4c Author: Valay Dave <[email protected]> Date: Sat Mar 19 22:58:37 2022 +0000 Changed retry setting. commit 563a20036a2dfcc48101f680f29d4917d53aa247 Author: Valay Dave <[email protected]> Date: Sat Mar 19 22:42:57 2022 +0000 Fixed setting `task_id` : - switch task-id from airflow job is to hash to "runid/stepname" - refactor xcom setting variables - added comments commit e2a1e502221dc603385263c82e2c068b9f055188 Author: Valay Dave <[email protected]> Date: Sat Mar 19 17:51:59 2022 +0000 setting retry logic. commit a697b56052210c8f009b68772c902bbf77713202 Author: Valay Dave <[email protected]> Date: Thu Mar 17 01:02:11 2022 +0000 Nit fix. commit 68f13beb17c7e73c0dddc142ef2418675a506439 Author: Valay Dave <[email protected]> Date: Wed Mar 16 20:46:19 2022 +0000 Added @schedule support + readme commit 57bdde54f9ad2c8fe5513dbdb9fd02394664e234 Author: Valay Dave <[email protected]> Date: Tue Mar 15 19:47:06 2022 +0000 Fixed setting run-id / task-id to labels in k8s - Fixed setting run-id has from cli macro - added hashing macro to ensure that jinja template set the correct run-id to k8s labels - commit 3d6c31917297d0be5f9915b13680fc415ddb4421 Author: Valay Dave <[email protected]> Date: Tue Mar 15 05:39:04 2022 +0000 Got linear workflows working on airflow. - Still not feature complete as lots of args are still unfilled / lots of unknows - minor tweek in eks to ensure airflow is k8s compatible. - passing state around via xcom-push - HACK : AWS keys are passed in a shady way. : Reverse this soon. commit db074b8012f76d9d85225a4ceddb2cde8fefa0f4 Author: Valay Dave <[email protected]> Date: Fri Mar 11 12:34:33 2022 -0800 Tweeks commit a9f0468c4721a2017f1b26eb8edcdd80aaa57203 Author: Valay Dave <[email protected]> Date: Tue Mar 1 17:14:47 2022 -0800 some changes based on savin's comments. - Added changes to task datastore for different reason : (todo) Decouple these - Added comments to SFN for reference. - Airflow DAG is no longer dependent on metaflow commit f32d089cd3865927bc7510f24ba3418d859410b6 Author: Valay Dave <[email protected]> Date: Wed Feb 23 00:54:17 2022 -0800 First version of dynamic dag compiler. - Not completely finished code - Creates generic .py file a JSON that is parsed to create Airflow DAG. - Currently only boiler plate to make a linear dag but doesn't execute anything. - Unfinished code. commit d2def665a86d6a6622d6076882c1c2d54044e773 Author: Valay Dave <[email protected]> Date: Sat Feb 19 14:01:47 2022 -0800 more tweeks. commit b176311f166788cc3dfc93354a0c5045a4e6a3d4 Author: Valay Dave <[email protected]> Date: Thu Feb 17 09:04:29 2022 -0800 commit 0 - unfinished code. * Making version compatibility changes. - Minimum support version to 2.2.0 * Task-id macro related logic refactor - Done for better version support. * bug fix: param related task-id setting. * applied black * Reverting `decorators.py` to master * Move sys.path insert earlier in s3op.py (#1098) * Update setup.py (#1099) * Fix for env_escape bug when importing local packages (#1100) * Bump to 2.7.5 (#1102) * Fix another issue with the escape hatch and paths (#1105) * Bump to 2.7.6 (#1106) * add a flag to overwrite config when running metaflow configure sandbox (#1103) * card dev docs tiny fix. (#1108) * Fix an issue with get_cards not respecting a Task's ds-root (#1111) get_cards would not always respect a Task's ds-root leading to cases where a Task has cards but they cannot be accessed because of an invalid path. * Adding support for Azure Blob Storage as a datastore (#1091) * Azure Storage * fix BrokenProcessPool import for older pythons * Fix batch * add azure configure, cannot validate storage account url after all (catch-22 in configure) * rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT * fix indent * clean up kubernetes config cli, add secrets * remove signature * fix fractional resoure handling for batch (#1089) * bump version (#1093) * Fix docstrings for the API reference (no functional changes!) (#1076) Make the docstring format in public APIs compliant with the new API reference framework * Fix issue with get_pinned_conda_libs and metaflow extensions * clean up includefile a bit, remove from_env (not used) * Move a sys.path modification in s3op to __main__ (#1095) In its current location, this could modify the sys.path of the current running metaflow which could have nefarious consequences with the escape hatch which uses sys.path to determine the outside environment's python path. The following scenario would cause issues: - metaflow is installed in the usual path on the system - a conda environment was manually bootstrapped from a directory A. + at this point, sys.path starts with `A` and then contains the other system includes + s3op.py is imported at some point by the Conda installer when it calls `get_many` + this modifies sys.path to insert, at the beginning, the parent of Metaflow; so in this case, sys.path looks something like ['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...] + when the escape hatch trampolines are created, this sys.path is used to determine what the sys.path for the outside interpreter is. + in A, we create: * INFO * metaflow * metaflow_extensions which properly describe the installation of metaflow - when the escape hatch client runs, it runs in the conda environment and uses metaflow created in A. - when the client wants to start the server, this is where we run into issues because, at this point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages' in which it will find metaflow. It will therefore use that metaflow (which is the same as the one linked in A) to start the server. This runs into issues though because A is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`. This will cause issues if multiple extensions are installed there (it will complain about duplicate configurations for example. The `INFO` file typically used to solve this problem is not read as it was not present for the TL metaflow. This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting sys.path when the module is simply included (and not called as a script). * Airflow Support (#1094) * Airflow on Kubernetes minus Foreachs. - Support for all metaflow construct without foreach and sensors Squashed commit of the following: commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1 Author: Valay Dave <[email protected]> Date: Fri Jul 29 01:06:26 2022 +0000 Removed sernsors and banned foreach's commit 8d517c4fecc6568777ad03eca81aaacfa3e91156 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:59:01 2022 +0000 commiting k8s related file from master. commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:54:45 2022 +0000 Uncommented code for foreach support with k8s KubernetesPodOperator version 4.2.0 renamed `resources` to `container_resources` - Check : (https://github.com/apache/airflow/pull/24673) / - (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85) This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator` and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. If it is > 4.2.0 then we set the argument as `container_resources` The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future. So we will only use it for `KuberentesPodOperator` version < 4.2.0 The `resources` argument will also not work for foreach's. commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:31:58 2022 +0000 nit fixes : - fixing comments. - refactor some variable/function names. commit 2079293fbba0d3d862476a7d67b36af8a3389342 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:14:53 2022 +0000 change `token` to `production_token` commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:11:56 2022 +0000 Refactored import Airflow Sensors. commit b1472d5f7a629024ca45e8b83700400d02a4d455 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:08:41 2022 +0000 new comment on `startup_timeout_seconds` env var. commit 6d81b758e8f06911258d26f790029f557488a0d7 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:06:09 2022 +0000 Removing traces of `@airflow_schedule_interval` commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946 Author: Valay Dave <[email protected]> Date: Thu Jul 14 12:43:08 2022 -0700 Foreach polish (valayDave/metaflow#62) * Removing unused imports * Added validation logic for airflow version numbers with foreaches * Removed `airflow_schedule_interval` decorator. * Added production/deployment token related changes - Uses s3 as a backend to store the production token - Token used for avoiding nameclashes - token stored via `FlowDatastore` * Graph type validation for airflow foreachs - Airflow foreachs only support single node fanout. - validation invalidates graphs with nested foreachs * Added configuration about startup_timeout. * Added final todo on `resources` argument of k8sOp - added a commented code block - it needs to be uncommented when airflow releasese the patch for the op - Code seems feature complete keeping aside airflow patch commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9 Author: Valay Dave <[email protected]> Date: Thu Jul 7 19:33:07 2022 +0000 Removed retries from user-defaults. commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43 Author: Valay Dave <[email protected]> Date: Wed Jul 6 16:29:33 2022 +0000 updated pod startup time commit fce2bd263f368dbb78a34ac71f64e13c89277222 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:11 2022 +0000 Adding default 1 retry for any airflow worker. commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321 Author: Valay Dave <[email protected]> Date: Mon Jun 27 01:22:42 2022 +0000 Airflow Foreach Integration - Simple one node foreach-join support as gaurenteed by airflow - Fixed env variable setting issue - introduced MetaflowKuberentesOperator - Created a new operator to allow smootness in plumbing xcom values - Some todos commit d319fa915c558d82f1d127736ce34d3ae0da521d Author: Valay Dave <[email protected]> Date: Fri Jun 24 21:12:09 2022 +0000 simplifying run-id macro. commit 0ffc813b1c4e6ba0103be51520f42d191371741a Author: Valay Dave <[email protected]> Date: Fri Jun 24 11:51:42 2022 -0700 Refactored parameter macro settings. (valayDave/metaflow#60) commit a3a495077f34183d706c0edbe56d6213766bf5f6 Author: Valay Dave <[email protected]> Date: Fri Jun 24 02:05:57 2022 +0000 added comment on need for `start_date` commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec Author: Valay Dave <[email protected]> Date: Tue Jun 21 06:03:56 2022 +0000 Refactored an `id_creator` method. commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:52:05 2022 +0000 refactor : -`RUN_ID_LEN` to `RUN_HASH_ID_LEN` - `TASK_ID_LEN` to `TASK_ID_HASH_LEN` commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2 Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:48:55 2022 +0000 refactored an error string commit 11458188b6c59d044fca0dd2d1f5024ec84f6488 Author: Valay Dave <[email protected]> Date: Mon Jun 20 22:42:36 2022 -0700 addressing savins comments. (#59) - Added many adhoc changes based for some comments. - Integrated secrets and `KUBERNETES_SECRETS` - cleaned up parameter setting - cleaned up setting of scheduling interval - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID` - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate` - Checking if dagfile and flow file are same. - fixing variable names. - checking out `kubernetes_decorator.py` from master (6441ed5) - bug fixing secret setting in airflow. - simplified parameter type parsing logic - refactoring airflow argument parsing code. commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646 Author: Valay Dave <[email protected]> Date: Mon Jun 13 14:02:57 2022 -0700 Addressing Final comments. (#57) - Added dag-run timeout. - airflow related scheduling checks in decorator. - Auto naming sensors if no name is provided - Annotations to k8s operators - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`) - annotation bug fix - setting`workflow-timeout` for only scheduled dags commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:50:49 2022 +0000 k8s bug fix commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:39:50 2022 +0000 removed un-used function commit 70e285e9a7cfbec71fc293508a62c96f33562a01 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:38:37 2022 +0000 Removed unused `sanitize_label` function commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:37:34 2022 +0000 GPU support added + container naming same as argo commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:25:17 2022 +0000 Refactored sensors to different files + bug fix - bug caused due `util.compress_list`. - The function doesn't play nice with strings with variety of characters. - Ensured that exceptions are handled appropriately. - Made new file for each sensor under `airflow.sensors` module. commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673 Author: Valay Dave <[email protected]> Date: Sat Jun 4 01:41:49 2022 +0000 ran black. commit 558c82f65b383ed0d61ded6bc80326471e284550 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:32:48 2022 -0700 Moving information from airflow_utils to compiler (#56) - commenting todos to organize unfinished changes. - some environment variables set via`V1EnvVar` - `client.V1ObjectFieldSelector` mapped env vars were not working in json form - Moving k8s operator import into its own function. - env vars moved. commit 9bb5f638792a671164ec95891e97f599e9a3385f Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:06:03 2022 +0000 added mising Run-id prefixes to variables. - merged `hash` and `dash_connect` filters. commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:00:22 2022 +0000 nit fix : variable name change. commit 660756f952ebd92ba1e26d7f908b81036c31ff10 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:58:34 2022 +0000 nit fixes to dag.py's templating variables. commit 1202f5bc92f76df52b5957f11c8574cadfa62196 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:56:53 2022 +0000 Fixed defaults passing - Addressed comments for airflow.py commit b9387dd428c1a37f9a3bfe2c72cab475da708c02 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:52:24 2022 +0000 Following Changes: - Refactors setting scheduling interval - refactor dag file creating function - refactored is_active to is_paused_upon_creation - removed catchup commit 054e3f389febc6c447494a1dedb01228f5f5650f Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:33:25 2022 +0000 Multiple Changes based on comments: 1. refactored `create_k8s_args` into _to_job 2. Addressed comments for snake casing 3. refactored `attrs` for simplicity. 4. refactored `metaflow_parameters` to `parameters`. 5. Refactored setting of `input_paths` commit d481b2fca7914b6b657a69af407cfe1a894a46dc Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:42:24 2022 +0000 Removed Sensor metadata extraction. commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3 Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:30:34 2022 +0000 porting savin's comments - next changes : addressing comments. commit 3f2353a647e53bc240e28792769c42a71ea8f8c9 Merge: d370ffb c1ff469 Author: Valay Dave <[email protected]> Date: Thu Jul 28 23:52:16 2022 +0000 Merge branch 'master' into airflow commit d370ffb248411ad4675f9d55de709dbd75d3806e Merge: a82f144 e4eb751 Author: Valay Dave <[email protected]> Date: Thu Jul 14 19:38:48 2022 +0000 Merge branch 'master' into airflow commit a82f1447b414171fc5611758cb6c12fc692f55f9 Merge: bdb1f0d 6f097e3 Author: Valay Dave <[email protected]> Date: Wed Jul 13 00:35:49 2022 +0000 Merge branch 'master' into airflow commit bdb1f0dd248d01318d4a493c75b6f54248c7be64 Merge: 8511215 f9a4968 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:51 2022 +0000 Merge branch 'master' into airflow commit 85112158cd352cb7de95a2262c011c6f43d98283 Author: Valay Dave <[email protected]> Date: Tue Jun 21 02:53:11 2022 +0000 Bug fix from master merge. commit 90c06f12bb14eda51c6a641766c5f67d6763abaa Merge: 0fb73af 6441ed5 Author: Valay Dave <[email protected]> Date: Mon Jun 20 21:20:20 2022 +0000 Merge branch 'master' into airflow commit 0fb73af8af9fca2875261e3bdd305a0daab1b229 Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:53:10 2022 +0000 squashing bugs after changes from master. commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d Merge: 7bdf662 ffff49b Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:20:38 2022 +0000 Merge branch 'master' into af-mmr commit 7bdf662e14966b929b8369c65d5bd3bbe5741937 Author: Valay Dave <[email protected]> Date: Mon May 16 17:42:38 2022 -0700 Airflow sensor api (#3) * Fixed run-id setting - Change gaurentees that multiple dags triggered at same moment have unique run-id * added allow multiple in `Decorator` class * Airflow sensor integration. >> support added for : - ExternalTaskSensor - S3KeySensor - SqlSensor >> sensors allow multiple decorators >> sensors accept those arguments which are supported by airflow * Added `@airflow_schedule_interval` decorator * Fixing bug run-id related in env variable setting. commit 2604a29452e794354cf4c612f48bae7cf45856ee Author: Valay Dave <[email protected]> Date: Thu Apr 21 18:26:59 2022 +0000 Addressed comments. commit 584e88b679fed7d6eec8ce564bf3707359170568 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:33:55 2022 +0000 fixed printing bug commit 169ac1535e5567149d94749ddaf70264e882d62c Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:30:59 2022 +0000 Option help bug fix. commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:25:54 2022 +0000 variable renamemetaflow_specific_args commit 0c779abcd1d9574878da6de8183461b53e0da366 Merge: d299b13 5a61508 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:23:10 2022 +0000 Merge branch 'airflow-tests' into airflow commit 5a61508e61583b567ef8d3fea04e049d74a6d973 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:22:54 2022 +0000 Removing un-used code / resolved-todos. commit d030830f2543f489a1c4ebd17da1b47942f041d6 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:06:03 2022 +0000 ran black, commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45 Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:04:19 2022 +0000 Merge branch 'master' into airflow-tests commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:02:37 2022 +0000 Merge branch 'master' into airflow commit f2cb3197725f11520da0d49cbeef8de215c243eb Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:54:03 2022 +0000 reverting change. commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4 Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:47:41 2022 +0000 3 changes: - Removing s3 dep - remove uesless import - added `deployed_on` in dag file template commit c6afba95f5ec05acf7f33fd3228cffd784556e3b Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:50:52 2022 +0000 Fixed passing secrets with kubernetes. commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:04:22 2022 +0000 Refactored code . - removed compute/k8s.py - Moved k8s code to airflow_compiler.py - ran isort to airflow_compiler.py commit d1c343dbbffbddbebd2aeda26d6846e595144e0b Author: Valay Dave <[email protected]> Date: Fri Apr 15 18:02:25 2022 +0000 Added validations about: - un-supported decorators - foreach Changed where validations are done to not save the package. commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce Author: Valay Dave <[email protected]> Date: Fri Apr 15 03:34:26 2022 +0000 Fixing mf log related bug - No double logging on metaflow. commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:10:51 2022 +0000 Removed usless code WRT project decorator. commit 5ad9a3949e351b0ac13f11df13446953932e8ffc Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:03:19 2022 +0000 Remove readme. commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:02:38 2022 +0000 Made file path required arguement. commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:01:07 2022 +0000 changed `--is-active`->`--is-paused-upon-creation` - dags are active by default. commit 5b98f937a62ee74de8aed8b0efde5045a28f068b Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:55:46 2022 +0000 shortened length of run-id and task-id hashes. commit e53426eaa4b156e8bd70ae7510c2e7c66745d101 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:41:32 2022 +0000 Remov…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bumps follow-redirects from 1.14.7 to 1.14.9.
Commits
13136e9Release version 1.14.9 of the npm package.2ec9b0bKeep headers when upgrading from HTTP to HTTPS.5fc74ddReduce nesting.3d81dc3Release version 1.14.8 of the npm package.62e546aDrop confidential headers across schemes.Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)@dependabot use these labelswill set the current labels as the default for future PRs for this repo and language@dependabot use these reviewerswill set the current reviewers as the default for future PRs for this repo and language@dependabot use these assigneeswill set the current assignees as the default for future PRs for this repo and language@dependabot use this milestonewill set the current milestone as the default for future PRs for this repo and languageYou can disable automated security fix PRs for this repo from the Security Alerts page.