Improve python-truenas-requirements doc (#13858)

docs/source/os/python.rst

@@ -4,41 +4,76 @@ Python
 .. contents:: Table of Contents
     :depth: 3
 
-Third-party python modules
+Third-party Python Modules
 --------------------------
 
-TrueNAS uses system-wide python environment and relies on Debian repository to install python packages. However, some
-of the required python packages are absent there, and others are outdated.
+TrueNAS uses a system-wide Python environment and relies on the Debian repository to install Python packages. However,
+some of the required Python packages are absent, and others are outdated.
 
-TrueNAS builder uses `python-truenas-requirements <https://github.com/truenas/python-truenas-requirements>`_ script to
-generate `deb` packages from `requirements.txt`. Each python package gets one corresponding `deb` package with the
-correct name, version and dependencies set; such packages integrate into TrueNAS Debian installation perfectly.
+The TrueNAS builder uses the python-truenas-requirements <https://github.com/truenas/python-truenas-requirements>_
+script to generate deb packages from requirements.txt. Each Python package gets one corresponding deb package with the
+correct name, version, and dependencies set; such packages integrate perfectly into the TrueNAS Debian installation.
 
-Adding a new third-party python module or upgrading an existing python module
+Adding a New Third-party Python Module or Upgrading an Existing Python Module
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-#. Add python module in question and all its dependencies into `requirements.txt`. Try to add as few as
-   possible; if one of the module's dependencies is shipped with Debian, its better to install it using apt.
-#. Run `generate.py` using the exact same Python version as TrueNAS. The easiest way to do this is using Docker:
+#. Add the Python module in question and all its dependencies to requirements.txt. Try to add as few as
+   possible; if one of the module's dependencies is shipped with Debian, it's better to install it using apt.
+
+   **Note 1**: If one of the dependencies is not added and it is absent or outdated in the apt repo, you'll get a clear
+   apt package install failure at the time of building the TrueNAS image, telling you which package is absent or
+   outdated.
+
+   **Note 2**: By default, Debian package names will be generated by prepending `python3-` to the PIP package name (for
+   example, `aiohttp` will become `python3-aiohttp`). However, some packages require special name handling (for example,
+   Debian ships `google-api-python-client` as `python3-googleapi`, not `python3-google-api-python-client`). Check the
+   `PIP_TO_DEBIAN_MAPPING` variable in the `generate.py` script to ensure the valid mapping is present for every such
+   package that is present in Debian to ensure we are correctly overriding them.
+#. Run `generate.py` using the exact same Python version as TrueNAS. The easiest way to do this is by using Docker:
 
     .. code-block:: shell
 
         ./generate.sh
 
-#. Ensure that the resulting diff is a small as possible. Use `constraints.txt` to pin down some indirect dependencies
+#. Ensure that the resulting diff is as small as possible. Use `constraints.txt` to pin down some indirect dependencies
    to the versions shipped with Debian.
 
-Upgrading all third-party python modules
+Upgrading All Third-party Python Modules
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Run `requirements-update.py` script to automatically update all items from `requirements.txt` to their latest versions.
+Run the `requirements-update.py` script to automatically update all items in `requirements.txt` to their latest
+versions.
 
-Removing unused python modules
+If an updated package version needs a new dependency that is not present or outdated in Debian, or any other dependency
+issues arise, you'll get a clear apt package install failure at the time of building the TrueNAS image, telling you
+exactly what the problem is.
+
+Removing Unused Python Modules
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Run `requirements-remove-unused.py` script **on the most recent TrueNAS build** to automatically remove
-`requirements.txt` entries that are were not used by that most recent build.
+Run the `requirements-remove-unused.py` script **on the most recent TrueNAS build** to automatically remove
+`requirements.txt` entries that were not used by that most recent build.
 
-This should be done when some of the python dependencies for TrueNAS components (middleware, CLI, etc.) were removed,
-or when apt mirrors were updated (providing official newer versions of the python packages we were previously building
+This should be done when some of the Python dependencies for TrueNAS components (middleware, CLI, etc.) were removed,
+or when apt mirrors were updated (providing official newer versions of the Python packages we were previously building
 ourselves).
+
+Why This Approach Was Taken
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following requirements were considered when designing this system:
+* Every PIP package should translate to an individual Debian package to allow seamless replacement of a Debian-shipped
+  package with a custom-built package and vice versa.
+* Adding/removing PIP packages to the system should be easy as we have many of them, and we have to conform to STIG
+  to quickly update packages with newly discovered CVEs without upgrading the whole Debian repo.
+* Builds should be stable; Python packages should never auto-update unless we update Debian repos or change
+  `requirements.txt` contents.
+
+These requirements led to the following design choices:
+
+* Every dependency (either direct or indirect) of every explicitly required package must be either already present
+  in the official Debian repos or be explicitly included in `requirements.txt`. This is the only way to pin those
+  dependencies to fixed versions to prevent them from auto-updating and breaking stable builds. Understandably,
+  these dependencies being present might seem redundant as they are not direct dependencies of our own Python
+  packages, but there will never be any unused packages there as the `requirements-remove-unused.py` script can easily
+  be used to remove them.