Improve provider documentation and README structure #32125

potiuk · 2023-06-25T21:29:33Z

The changelong in provider docs has been part of the README which became part of the page displayed in PyPI as well as the main page displayed when you want to see provider documentation. This was fine when the changelog was short, but with providers history growing, the main page became longer and longer and the generated index on the right side (from the changelog) became less and less usable, and changelog itself became far less discoverable.

This change introduces another approach. Changelog gets its own, separate page and it has quite a few positive consequences:

we can add separate "Changelog" URL in PyPI so that it is
prominently visible at the TOC in PyPI URLS
we can link to the changelog easily from stable page - without
worrrying which version we link to
the README file is part of the package so fixing the changelog
in case of a doc-only mistake will not fix the PyPI page (it
will only be fixed in future releases)
the link tree on the right side of the provider index page becomes back
usable
the left-side TOD on the provider's index page containes two
prominent links that were missing: Home (back to the index provider
page), Changelog, and Security - all under "Basics" link
The TOC is not repeated in the index page itself, making the content
immediately visible rather than having to scroll trough the long
TOC (in some cases like amazon the TOC was reallly long).

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-06-25T21:32:24Z

cc: @eladkal. @pierrejeambrun, @ephraimbuddy much better provider packages docs:

Main page:

No changelog in the main page:

Separate changelog page:

Seperate security page:

potiuk · 2023-06-25T21:33:26Z

Compare to original page:

dev/provider_packages/PROVIDER_CHANGELOG_TEMPLATE.rst.jinja2

airflow/providers/elasticsearch/CHANGELOG.rst

potiuk · 2023-06-26T06:50:35Z

BTW. Once this is merged I will also add redirection to "index" for historical versions in https://github.com/apache/airflow-site/blob/main/post-docs/add-back-references.py and post-process older versions to contain the redirects.

dev/provider_packages/PROVIDER_CHANGELOG_TEMPLATE.rst.jinja2

pankajkoti

Nice one, some minor questions/comments inline.

airflow/providers/elasticsearch/__init__.py

dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2

docs/conf.py

dev/provider_packages/PROVIDER_CHANGELOG_TEMPLATE.rst.jinja2

potiuk · 2023-06-26T10:26:38Z

Applied next round of comments. Also added "allow suspended providers changes" as this one also includes yandex command.

airflow/providers/elasticsearch/provider.yaml

pankajkoti · 2023-06-26T10:57:36Z

dev/provider_packages/prepare_provider_packages.py

---------
+
+Changelog for ``{{ package_name }}``
+{{ '-' * (18 + package_name | length) }}


Nice 👍🏽

I just feel sometimes that GitHub could provide adding emojis to react on lines while reviewing.

eladkal · 2023-06-26T15:16:14Z

docs/apache-airflow-providers-apache-kafka/security.rst

+Releasing security patches
+--------------------------
+
+Airflow providers are released independently from Airflow itself and the information about vulnerabilities
+is published separately. You can upgrade providers independently from Airflow itself, following the
+instructions found in :doc:`apache-airflow:installation/installing-from-pypi`.
+
+When we release Provider version, the development is always done from the ``main`` branch where we prepare
+the next version. The provider uses strict `SemVer <https://semver.org>`_ versioning policy. Depending on
+the scope of the change, Provider will get ''MAJOR'' version upgrade when there are
+breaking changes, ``MINOR`` version upgrade when there are new features or ``PATCHLEVEL`` version upgrade
+when there are only bug fixes (including security bugfixes) - and this is the only version that receives
+security fixes by default, so you should upgrade to latest version of the provider if you want to receive
+all released security fixes.
+
+The only exception to that rule is when we have a critical security fix and good reason to provide an
+out-of-band release for the provider, in which case stakeholders in the provider might decide to cherry-pick
+and prepare a branch for an older version of the provider following the
+`mixed governance model <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#mixed-governance-model>`_
+and requires interested parties to cherry-pick and test the fixes.


This is a repeated text for all the providers.
Maybe we should extract this text to
docs/apache-airflow-providers/security.rst and just share a link here?
The issue is that every edit to this content will force handling doc release for all providers.

This was a deliberate choice, actually. I think we want a copy of security page for each provider.

This text will be updated when we will publish SBOM information with links to SBOM information, so the content of this one will be different for every provider anyway (It has no variable part but it will be coming).

It's very similar to those pages:

https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/installing-providers-from-sources.html

Where links to .sha/downloaded tar.gz are generated automatically when documentation is published, but other than that the content will be the same.

I think it is important to have this information "in-provider" separately for each provider so that you do not have to go "elsewhere" to find the information and follow some "generic" advices - just to link directly to the information you need.

The issue is that every edit to this content will force handling doc release for all providers.

Correct. This is how it should be really, it's handled by "doc-only changes" option - this one will not release provider itself but it will update the information in the latest released version of the provider. And once we add those links (I hope it will be before the next wave) we will have to updated them for all providers anyway (possibly even without actually releasing the providers because SBOM information is "post-processing" the releases - does not have to cause a new provider release. I am actually even planning to retroactively add such generated "security" pages back in all released version of the provider docs so that the user can see the SBOM information even in old versions of the docs.

One more comment - for me this s really a matter of UI/UX - simply speakiing - in documenation, there is often a value when the same information is available directly when you look for it rather than having to do "one-more-click" - we need such security page "per-provider" anyway. We could extract the common part/text to a separate document and link to it of course. But it makes it "one-more-click-away" - i.e many peopke wont make the effort to follow the link.

This was a deliberate choice

this explanation is enough for me :)

The changelong in provider docs has been part of the README which became part of the page displayed in `PyPI` as well as the main page displayed when you want to see provider documentation. This was fine when the changelog was short, but with providers history growing, the main page became longer and longer and the generated index on the right side (from the changelog) became less and less usable, and changelog itself became far less discoverable. This change introduces another approach. Changelog gets its own, separate page and it has quite a few positive consequences: 1) we can add separate "Changelog" URL in PyPI so that it is prominently visible at the TOC in PyPI URLS 2) we can link to the changelog easily from stable page - without worrrying which version we link to 3) the README file is part of the package so fixing the changelog in case of a doc-only mistake will not fix the PyPI page (it will only be fixed in future releases) 4) the link tree on the right side of the provider index page becomes back usable 5) the left-side TOD on the provider's index page containes two prominent links that were missing: Home (back to the index provider page), Changelog, and Security - all under "Basics" link 6) The TOC is not repeated in the index page itself, making the content immediately visible rather than having to scroll trough the long TOC (in some cases like amazon the TOC was reallly long).

potiuk requested review from ashb, jedcunningham, eladkal, o-nikolas, Fokko, josh-fell, mik-laj and kaxil as code owners June 25, 2023 21:29

boring-cyborg bot added provider:cncf-kubernetes Kubernetes provider related issues area:providers provider:Apache provider:celery provider:common-sql provider:databricks labels Jun 25, 2023

potiuk requested review from utkarsharma2, ephraimbuddy and pierrejeambrun June 25, 2023 21:29

potiuk requested review from pankajkoti, pankajastro and phanikumv and removed request for utkarsharma2 June 25, 2023 21:36

eladkal reviewed Jun 26, 2023

View reviewed changes

dev/provider_packages/PROVIDER_CHANGELOG_TEMPLATE.rst.jinja2 Show resolved Hide resolved

airflow/providers/elasticsearch/CHANGELOG.rst Outdated Show resolved Hide resolved

potiuk force-pushed the move-changelogs-in-provider-docs branch 2 times, most recently from 2946f87 to e8afb0d Compare June 26, 2023 06:47

uranusjr reviewed Jun 26, 2023

View reviewed changes

dev/provider_packages/PROVIDER_CHANGELOG_TEMPLATE.rst.jinja2 Outdated Show resolved Hide resolved

eladkal mentioned this pull request Jun 26, 2023

set contributor note in prepare_provider_packages.py #32019

Closed

pankajkoti reviewed Jun 26, 2023

View reviewed changes

potiuk force-pushed the move-changelogs-in-provider-docs branch from e8afb0d to f7de112 Compare June 26, 2023 10:25

potiuk added the allow suspended provider changes Allow changes in suspended providers label Jun 26, 2023

potiuk force-pushed the move-changelogs-in-provider-docs branch from f7de112 to 5c2356f Compare June 26, 2023 10:26

pankajkoti approved these changes Jun 26, 2023

View reviewed changes

potiuk force-pushed the move-changelogs-in-provider-docs branch from 5c2356f to 444bdca Compare June 26, 2023 11:27

eladkal reviewed Jun 26, 2023

View reviewed changes

eladkal approved these changes Jun 27, 2023

View reviewed changes

potiuk force-pushed the move-changelogs-in-provider-docs branch from 444bdca to 873c778 Compare June 27, 2023 12:40

potiuk merged commit 09d4718 into apache:main Jun 27, 2023

potiuk deleted the move-changelogs-in-provider-docs branch June 27, 2023 17:05

ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jul 6, 2023

This was referenced Jul 29, 2023

Status of testing Providers that were prepared on July 29, 2023 #32936

Closed

Status of testing Providers that were prepared on August 05, 2023 #33132

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve provider documentation and README structure #32125

Improve provider documentation and README structure #32125

potiuk commented Jun 25, 2023

potiuk commented Jun 25, 2023

potiuk commented Jun 25, 2023

potiuk commented Jun 26, 2023

pankajkoti left a comment

potiuk commented Jun 26, 2023

pankajkoti Jun 26, 2023

eladkal Jun 26, 2023 •

edited

Loading

potiuk Jun 27, 2023

potiuk Jun 27, 2023 •

edited

Loading

eladkal Jun 27, 2023 •

edited

Loading

Improve provider documentation and README structure #32125

Improve provider documentation and README structure #32125

Conversation

potiuk commented Jun 25, 2023

potiuk commented Jun 25, 2023

potiuk commented Jun 25, 2023

potiuk commented Jun 26, 2023

pankajkoti left a comment

Choose a reason for hiding this comment

potiuk commented Jun 26, 2023

pankajkoti Jun 26, 2023

Choose a reason for hiding this comment

eladkal Jun 26, 2023 • edited Loading

Choose a reason for hiding this comment

potiuk Jun 27, 2023

Choose a reason for hiding this comment

potiuk Jun 27, 2023 • edited Loading

Choose a reason for hiding this comment

eladkal Jun 27, 2023 • edited Loading

Choose a reason for hiding this comment

eladkal Jun 26, 2023 •

edited

Loading

potiuk Jun 27, 2023 •

edited

Loading

eladkal Jun 27, 2023 •

edited

Loading