Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve back-reference generation #33149

Merged
merged 1 commit into from
Aug 6, 2023

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Aug 6, 2023

  • better command line support with better auto-complete
  • allows to run back-referance generation for individual provider packages
  • better typing including using Pathlib everywhere
  • handling of redirection between providers - when an operator is moved from one provider to the other - instead of generating invalid "stable" link we generate back reference for all versions of the provider.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

* better command line support with better auto-complete
* allows to run back-referance generation for individual provider packages
* better typing including using Pathlib everywhere
* handling of redirection between providers - when an operator is
  moved from one provider to the other - instead of generating invalid
  "stable" link we generate back reference for all versions of the
  provider.
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impressive work overall. Can see a few small nits

Copy link
Contributor

@eladkal eladkal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@potiuk potiuk merged commit 14faf19 into apache:main Aug 6, 2023
@potiuk potiuk deleted the remove-stable-backreference branch August 6, 2023 12:28
@potiuk
Copy link
Member Author

potiuk commented Aug 6, 2023

You should be able to re-apply it now to your PR - it should add a bunch of redirections for old google provider versions for the microsoft.azure redirection.. Also You can also try to run it for microsoft.azure only - but I think just re-running for all should work.

@eladkal
Copy link
Contributor

eladkal commented Aug 8, 2023

@potiuk I think something is wrong.

Running:
breeze release-management add-back-references --gen-type providers

Also scan non-provider refs.

Example:

Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/apache-airflow-providers-zendesk, version: 3.0.3
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/apache-airflow-providers-zendesk/3.0.3/changelog.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/apache-airflow-providers-zendesk/3.0.3/security.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/apache-airflow-providers-zendesk/3.0.3/security/security-model.html, redirects already exist
Processing airflow provider: docker-stack
The https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/redirects.txt does not exist. Skipping.
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack, version: _modules
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack/_modules/changelog.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack/_modules/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack/_modules/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack, version: _sources
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack, version: _static
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack, version: _images
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/docker-stack, version: output-spelling-results-docker-stack
Processing airflow provider: helm-chart
Constructs old to new mapping from redirects.txt for /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.9.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.9.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.9.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.9.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.9.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.9.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.3.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.3.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.3.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.3.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.3.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.3.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.7.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.7.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.7.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.7.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.7.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.7.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.0.0
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.0.0/changelog.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.0.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.0.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.4.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.4.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.4.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.4.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.4.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.4.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.1.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.1.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.1.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.1.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.1.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.1.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.5.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.5.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.5.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.5.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.5.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.5.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.10.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.10.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.10.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.10.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.10.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.10.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.2.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.2.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.2.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.2.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.2.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.2.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.8.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.8.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.8.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.8.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.8.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.8.0/security/security-model.html
Processing /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart, version: 1.6.0
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.6.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.6.0/release_notes.html, redirects already exist
Skipping file:/Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.6.0/changelog.html, redirects already exist
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.6.0/security.html
Created back reference redirect: /Users/eladkal/PycharmProjects/airflow-site/docs-archive/helm-chart/1.6.0/security/security-model.html

@potiuk
Copy link
Member Author

potiuk commented Aug 8, 2023

This is fine. It will skip all the the things that are not "versions".

@eladkal
Copy link
Contributor

eladkal commented Aug 8, 2023

This is fine. It will skip all the the things that are not "versions".

but it doesn't..
it added changes to airflow core, helm chart and others...
I had to manually remove the changes before opening the PR for doc site

@potiuk
Copy link
Member Author

potiuk commented Aug 8, 2023

Final (hopefully) fix - removing the need of --gen-type and resulting ambiguities in #33207

@ephraimbuddy ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Aug 8, 2023
@ephraimbuddy ephraimbuddy added this to the Airflow 2.7.0 milestone Aug 8, 2023
potiuk added a commit to potiuk/airflow that referenced this pull request Aug 8, 2023
This is a better fix for add-back-references breeze command, as
the previous fix had a bug and processed more than just providers
when providers --gen-type was selected. See apache#33149.

This version fixes it in a much more consistent way:

* you have to specify what you want to work on as option
* you can  select few of the provider via auto-complete
* you can also select apache-airflow and helm-chart
* you can mix and match them if you need

All the scripts and references where `--gen-type` was used
previously have been updated and documentation was changed to
reflect it
ephraimbuddy pushed a commit that referenced this pull request Aug 8, 2023
* better command line support with better auto-complete
* allows to run back-referance generation for individual provider packages
* better typing including using Pathlib everywhere
* handling of redirection between providers - when an operator is
  moved from one provider to the other - instead of generating invalid
  "stable" link we generate back reference for all versions of the
  provider.

(cherry picked from commit 14faf19)
potiuk added a commit that referenced this pull request Aug 9, 2023
This is a better fix for add-back-references breeze command, as
the previous fix had a bug and processed more than just providers
when providers --gen-type was selected. See #33149.

This version fixes it in a much more consistent way:

* you have to specify what you want to work on as option
* you can  select few of the provider via auto-complete
* you can also select apache-airflow and helm-chart
* you can mix and match them if you need

All the scripts and references where `--gen-type` was used
previously have been updated and documentation was changed to
reflect it
ephraimbuddy pushed a commit that referenced this pull request Oct 5, 2023
* better command line support with better auto-complete
* allows to run back-referance generation for individual provider packages
* better typing including using Pathlib everywhere
* handling of redirection between providers - when an operator is
  moved from one provider to the other - instead of generating invalid
  "stable" link we generate back reference for all versions of the
  provider.

(cherry picked from commit 14faf19)
ephraimbuddy pushed a commit that referenced this pull request Oct 5, 2023
This is a better fix for add-back-references breeze command, as
the previous fix had a bug and processed more than just providers
when providers --gen-type was selected. See #33149.

This version fixes it in a much more consistent way:

* you have to specify what you want to work on as option
* you can  select few of the provider via auto-complete
* you can also select apache-airflow and helm-chart
* you can mix and match them if you need

All the scripts and references where `--gen-type` was used
previously have been updated and documentation was changed to
reflect it

(cherry picked from commit 8c28454)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools type:misc/internal Changelog: Misc changes that should appear in change log
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants