-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This adds a `--prune` flag to the `export` command to correspond with the `--prune` flag of the `tree` command. The purpose is for generating a `requirements.txt` that omits a package and all of that package's unique dependencies. This is useful for cases where the project has a dependency on a common core package, but where that package does not need to be installed in the target environment. For example, a pyspark job needs spark for development, but when installing into a cluster that already has pyspark installed, it is desirable to omit pyspark's whole dependency tree so that only the unique dependencies that your job needs get installed, and do not risk breaking the pyspark dependencies with something incompatible. Dev groups cannot always cover this case because there are other projects where this common dependency occurs as a transitive. One example is Airflow providers, which include Airflow itself as a dependency, but it is unnecessary and undesirable to include Airflow's dependency tree in the `requirements.txt` for your DAGs.
- Loading branch information
Showing
7 changed files
with
83 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters