Build: install all the latest Python "core requirements"

I created a new feature flag called `INSTALL_LATEST_CORE_REQUIREMENTS` that does
not depend on any other feature flag. Its goal is to always install the latest
Python packages.

We are trying to move away from pinning the dependencies on our side and telling
users to do that on their side if they want to have reproducible builds over
time.

I think this is the best outcome for new projects since they will immediately
use the latest versions of everything instead of old (and maybe broken)
dependencies.

I propose to set a particular date for this feature flag and make new projects
to start building with all the latest dependencies. This will, at least, stop
making the problem bigger. Later, we can decide how to communicate to old
projects that we are going to install the latest requirements and if they don't
want that, they should pin their dependencies. We can follow our new and shiny
deprecation path we have built and tested in the last month.

Related readthedocs/meta#8
Related #9562
Related #10402
Related #9081

readthedocs/doc_builder/python_environments.py

@@ -163,6 +163,67 @@ def install_core_requirements(self):
             '--no-cache-dir',
         ]
 
+        if self.project.has_feature(Feature.INSTALL_LATEST_CORE_REQUIREMENTS):
+            self._install_latest_requirements(pip_install_cmd)
+        else:
+            self._install_old_requirements(pip_install_cmd)
+
+    def _install_latest_requirements(self, pip_install_cmd):
+        """
+        Install all the latest core requirements.
+
+        By enabling the feature flag ``INSTALL_LATEST_CORE_REQUIREMENTS``
+        projects will automatically get installed all the latest core
+        requirements: pip, sphinx, mock, alabaster, setuptools, etc.
+
+        This is the new behavior and where we are moving towards.
+        """
+        # First, upgrade pip and setuptools to their latest versions
+        cmd = pip_install_cmd + ["pip", "setuptools"]
+        self.build_env.run(
+            *cmd,
+            bin_path=self.venv_bin(),
+            cwd=self.checkout_path,
+        )
+
+        # Second, install all the latest core requirements
+        requirements = [
+            "mock",
+            "alabaster",
+            "commonmark",
+            "recommonmark",
+            "setuptools",
+        ]
+
+        if self.config.doctype == "mkdocs":
+            requirements.append("mkdocs")
+        else:
+            requirements.extend(
+                [
+                    "sphinx",
+                    "sphinx-rtd-theme",
+                    "readthedocs-sphinx-ext",
+                ]
+            )
+
+        cmd = copy.copy(pip_install_cmd)
+        cmd.extend(requirements)
+        self.build_env.run(
+            *cmd,
+            bin_path=self.venv_bin(),
+            cwd=self.checkout_path,
+        )
+
+    def _install_old_requirements(self, pip_install_cmd):
+        """
+        Install old core requirements.
+
+        There are bunch of feature flags that will be taken in consideration to
+        decide whether or not upgrade some of the core dependencies to their
+        latest versions.
+
+        This is the old behavior and the one we want to get rid off.
+        """
         # Install latest pip and setuptools first,
         # so it is used when installing the other requirements.
         pip_version = self.project.get_feature_value(

readthedocs/projects/models.py

@@ -1930,6 +1930,7 @@ def add_features(sender, **kwargs):
     DEFAULT_TO_MKDOCS_0_17_3 = 'default_to_mkdocs_0_17_3'
     USE_MKDOCS_LATEST = 'use_mkdocs_latest'
     USE_SPHINX_RTD_EXT_LATEST = 'rtd_sphinx_ext_latest'
+    INSTALL_LATEST_CORE_REQUIREMENTS = "install_latest_core_requirements"
 
     # Search related features
     DISABLE_SERVER_SIDE_SEARCH = 'disable_server_side_search'
@@ -2044,6 +2045,12 @@ def add_features(sender, **kwargs):
             USE_SPHINX_RTD_EXT_LATEST,
             _("Sphinx: Use latest version of the Read the Docs Sphinx extension"),
         ),
+        (
+            INSTALL_LATEST_CORE_REQUIREMENTS,
+            _(
+                "Build: Install all the latest versions of Read the Docs core requirements"
+            ),
+        ),
 
         # Search related features.
         (