Protect against manually updated generated dependencies #37052
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
commented
Jan 28, 2024
potiuk
force-pushed
the
protect-against-not-regenerated-pyproject-toml
branch
from
cde5867 to
e784cd5
Compare
|
I also added a small thing - in the generated pyproject.toml I've added comment stating where the source of the information is - pointing to the provider.yaml file where the dependencies are coming from. I hope that will avoid cases where people will try to modify this part manually. |
Taragolis
approved these changes
Jan 28, 2024
The "generated/provider_dependencies.json" is regenerated automatically together with pyproject.toml from all the provider.yaml files. The file contains information about dependencies for all providers and it is used to determine a number of actions when we build providers (for example it is used to automatically generated cross-provider dependencies, and determine whether provider is ready to be released) The "dependencies" from provider_dependencies.json are also reflected in pyproject.toml file in order to determine what should be installed when you install specific editable extra, so pyproject.toml is also generated automatically together with provider_dependencies.json. All is good when that generation is done automatically, but so far, when you updated provider_dependencies.json manually, the pyproject.toml was not regenerated - it was actually skipped from regeneration. This PR changes it by storing hash of provider_depenedencies.json (mixed with the has of the script that generates them) in the generated pyproject.toml file. This way, both - provider_dependencies and pyproject.toml wil always be regenerated by the "update_providers_dependencies.py" pre-commit whenever there is any inconsistency between provider.yaml files, provider_dependencies.json or pyproject.toml.
potiuk
force-pushed
the
protect-against-not-regenerated-pyproject-toml
branch
from
e784cd5 to
6660cc0
Compare
Oh. Yeah. I really, really like seeing someone problems/struggling and turning it into a solution to prevent them. |
|
AH, I need to close that one and open one from the |
|
All right. I actually found a way, how to avoid having to do that again: #37057 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The "generated/provider_dependencies.json" is regenerated automatically together with pyproject.toml from all the provider.yaml files. The file contains information about dependencies for all providers and it is used to determine a number of actions when we build providers (for example it is used to automatically generated cross-provider dependencies, and determine whether provider is ready to be released)
The "dependencies" from provider_dependencies.json are also reflected in pyproject.toml file in order to determine what should be installed when you install specific editable extra, so pyproject.toml is also generated automatically together with
provider_dependencies.json. All is good when that generation is done automatically, but so far, when you updated provider_dependencies.json manually, the pyproject.toml was not regenerated - it was actually skipped from regeneration.
This PR changes it by storing hash of provider_depenedencies.json (mixed with the has of the script that generates them) in the generated pyproject.toml file. This way, both - provider_dependencies and pyproject.toml wil always be regenerated by the
"update_providers_dependencies.py" pre-commit whenever there is any inconsistency between provider.yaml files, provider_dependencies.json or pyproject.toml.
Related: #37040
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.