feat: upgrade to Maple

- A shared cookie domain between lms and cms is no longer recommended: https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst - refactor: clean mounted data folder in lms/cms. In Lilac, the bind-mounted lms/data and cms/data folders are a mess because new folders are created there for every new course organisation. These folders are empty. As far as we know they are useless... With this change we move these folders to a dedicated "modulestore" subdirectory; which corresponds better to the initial intent of the fs_root setting. - fix: frontend failure during login to the lms. See: openedx/wg-build-test-release#104 - feat: move all forum-related code to a dedicated plugin. Forum is an optional feature, and as such it deserves its own plugin. Starting from Maple, users will be able to install the forum from https://github.com/overhangio/tutor-forum/ - migrate from DCS_* session cookie settings to SESSION_*. That's because edx-platform no longer depends on django-cookies-samesite. Close openedx/wg-build-test-release#110 - get rid of tons of deprecation warnings in the lms/cms - feat: make it possible to point to themed assets. Cherry-picking this change makes it possible to point to themed assets with a theme-agnostic url, notably from MFEs. Close #450.
overhangio · Dec 16, 2021 · 1e019d8 · 1e019d8
1 parent 1fe311d
commit 1e019d8
Show file tree

Hide file tree

Showing 28 changed files with 161 additions and 85 deletions.
diff --git a/CHANGELOG-nightly.md b/CHANGELOG-nightly.md
@@ -1,26 +1,3 @@
 # Changelog (nightly branch)
 
 Note: Breaking changes between versions are indicated by "💥".
-
-- [Feature] Better support of Caddy as a load balancer in Kubernetes:
-  - Make it possible to start/stop a selection of resources with ``tutor k8s start/stop [names...]``.
-  - Make it easy to deploy an independent LoadBalancer by converting the caddy service to a ClusterIP when ``ENABLE_WEB_PROXY=false``.
-  - Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer service.
-  - Add ``app.kubernetes.io/name`` labels to all services.
-  - Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
-  - Wait for the caddy deployment to be ready before running initialisation jobs.
-- [Security] On Kubernetes, convert all NodePort services to ClusterIP to guarantee network isolation from outside the cluster.
-- 💥[Improvement] Drop Python 3.5 compatibility.
-- [Bugfix] Fix docker-compose project name in development on nightly branch.
-- 💥[Bugfix] No longer track the Tutor version number in resource labels (and label selectors, which breaks the update of Deployment resources), but instead do so in resource annotations.
-- [Bugfix] Make it possible for plugins to implement the "caddyfile" patch without relying on the "port" local variable.
-- 💥[Improvement] Move the Open edX forum to a [dedicated plugin](https://github.com/overhangio/tutor-forum/) (#450).
-- 💥[Improvement] Get rid of the "tutor-openedx" package, which is no longer supported.
-- [Bugfix] Fix running Caddy container in k8s, which should always be the case even if `ENABLE_WEB_PROXY` is false.
-- 💥[Improvement] Run all services as unprivileged containers, for better security. This has multiple consequences:
-  - The "openedx-dev" image is now built with `tutor dev dc build lms`.
-  - The "smtp" service now runs the "devture/exim-relay" Docker image, which is unprivileged. Also, the default SMTP port is now 8025.
-- 💥[Feature] Get rid of the nginx container and service, which is now replaced by Caddy. this has the following consequences:
-    - Patches "nginx-cms", "nginx-lms", "nginx-extra", "local-docker-compose-nginx-aliases" are replaced by "caddyfile-cms", "caddyfile-lms", "caddyfile", " local-docker-compose-caddy-aliases".
-    - Patches "k8s-deployments-nginx-volume-mounts", "k8s-deployments-nginx-volumes" were obsolete and are removed.
-    - The `NGINX_HTTP_PORT` setting is renamed to `CADDY_HTTP_PORT`.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,27 @@ Note: Breaking changes between versions are indicated by "💥".
 
 ## Unreleased
 
+- 💥[Improvement] Upgrade to Maple
+- [Feature] Better support of Caddy as a load balancer in Kubernetes:
+  - Make it possible to start/stop a selection of resources with ``tutor k8s start/stop [names...]``.
+  - Make it easy to deploy an independent LoadBalancer by converting the caddy service to a ClusterIP when ``ENABLE_WEB_PROXY=false``.
+  - Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer service.
+  - Add ``app.kubernetes.io/name`` labels to all services.
+  - Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
+  - Wait for the caddy deployment to be ready before running initialisation jobs.
+  - Fix running Caddy container in k8s, which should always be the case even if `ENABLE_WEB_PROXY` is false.
+- [Security] On Kubernetes, convert all NodePort services to ClusterIP to guarantee network isolation from outside the cluster.
+- 💥[Improvement] Move the Open edX forum to a [dedicated plugin](https://github.com/overhangio/tutor-forum/) (#450).
+- 💥[Improvement] Drop Python 3.5 compatibility.
+- 💥[Bugfix] No longer track the Tutor version number in resource labels (and label selectors, which breaks the update of Deployment resources), but instead do so in resource annotations.
+- 💥[Improvement] Get rid of the "tutor-openedx" package, which is no longer supported.
+- 💥[Improvement] Run all services as unprivileged containers, for better security. This has multiple consequences:
+  - The "openedx-dev" image is now built with `tutor dev dc build lms`.
+  - The "smtp" service now runs the "devture/exim-relay" Docker image, which is unprivileged. Also, the default SMTP port is now 8025.
+- 💥[Feature] Get rid of the nginx container and service, which is now replaced by Caddy. this has the following consequences:
+    - Patches "nginx-cms", "nginx-lms", "nginx-extra", "local-docker-compose-nginx-aliases" are replaced by "caddyfile-cms", "caddyfile-lms", "caddyfile", " local-docker-compose-caddy-aliases".
+    - Patches "k8s-deployments-nginx-volume-mounts", "k8s-deployments-nginx-volumes" were obsolete and are removed.
+    - The `NGINX_HTTP_PORT` setting is renamed to `CADDY_HTTP_PORT`.
 - [Bugfix] Fix building of the `openedx` image on ARM64 due to missing `libgeos-dev`
 
 ## v12.2.0 (2021-12-08)

diff --git a/docs/configuration.rst b/docs/configuration.rst
@@ -79,7 +79,7 @@ You may want to pull/push images from/to a custom docker registry. For instance,
 Open edX customisation
 ~~~~~~~~~~~~~~~~~~~~~~
 
-- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/lilac.2"``)
+- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/maple.beta2"``)
 
 This defines the default version that will be pulled from all Open edX git repositories.
 
@@ -281,16 +281,16 @@ Note that your edx-platform version must be a fork of the latest release **tag**
 
 If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular:
 
-- Do not try to run a fork from an older (pre-Lilac) version of edx-platform: this will simply not work.
+- Do not try to run a fork from an older (pre-Maple) version of edx-platform: this will simply not work.
 - Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail.
-- Do not try to run a fork from the open-release/lilac.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/lilac.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/lilac.master branch.
+- Do not try to run a fork from the open-release/maple.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/maple.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/maple.master branch.
 
 .. _i18n:
 
 Adding custom translations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX in your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/edx/edx-platform/tree/open-release/lilac.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
+If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX in your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/edx/edx-platform/tree/open-release/maple.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
 
 Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance::
 
@@ -311,9 +311,9 @@ Then, add a "django.po" file there that will contain your custom translations::
 .. warning::
     Don't forget to specify the file ``Content-Type`` when adding message strings with non-ASCII characters; otherwise a ``UnicodeDecodeError`` will be raised during compilation.
 
-The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/edx/edx-platform/blob/open-release/lilac.master/conf/locale/en/LC_MESSAGES/django.po>`__.
+The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/edx/edx-platform/blob/open-release/maple.master/conf/locale/en/LC_MESSAGES/django.po>`__.
 
-If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/edx/edx-platform/blob/open-release/lilac.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
+If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/edx/edx-platform/blob/open-release/maple.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
 
 To recap, here is an example. To translate a few strings in French, both from django.po and djangojs.po, we would have the following file hierarchy::
 

diff --git a/docs/dev.rst b/docs/dev.rst
@@ -25,7 +25,7 @@ This ``openedx-dev`` development image differs from the ``openedx`` production i
 
 - The user that runs inside the container has the same UID as the user on the host, in order to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository).
 - Additional python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
-- The edx-platform `development requirements <https://github.com/edx/edx-platform/blob/open-release/lilac.master/requirements/edx/development.in>`__ are installed.
+- The edx-platform `development requirements <https://github.com/edx/edx-platform/blob/open-release/maple.master/requirements/edx/development.in>`__ are installed.
 
 Since the ``openedx-dev`` is based upon the ``openedx`` docker image, it should be re-built every time the ``openedx`` docker image is modified.
 
@@ -137,7 +137,7 @@ Following the instructions :ref:`above <bind_mounts>` on how to bind-mount direc
 
 If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
 
-First of all, you should make sure that you are working off the ``open-release/lilac.2`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
+First of all, you should make sure that you are working off the ``open-release/maple.beta2`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
 
 Then, you should run the following commands::
 

diff --git a/docs/faq.rst b/docs/faq.rst
@@ -38,7 +38,7 @@ The `devstack <https://github.com/edx/devstack>`_ is meant for development only,
 Is Tutor officially supported by edX?
 -------------------------------------
 
-As of the Open edX Lilac release (June 9th 2021), Tutor is one of the two officially supported installation methods for Open edX: see the `official installation instructions <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-lilac.master/installation/index.html>`__. We expect that by Maple (December 9th 2021) the native installation will be deprecated and Tutor will become the only officially recommended installation method, unless major issues are discovered. However, Tutor remains developed independently from edX, both by its parent company Overhang.IO and the :ref:`project maintainers <maintainers>`.
+Yes: as of the Open edX Maple release (December 9th 2021), Tutor is the only officially supported installation methods for Open edX: see the `official installation instructions <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-maple.master/installation/index.html>`__. 
 
 What features are missing from Tutor?
 -------------------------------------

diff --git a/docs/install.rst b/docs/install.rst
@@ -86,9 +86,9 @@ Upgrading
 
 With Tutor, it is very easy to upgrade to a more recent Open edX or Tutor release. Just install the latest ``tutor`` version (using either methods above) and run the ``quickstart`` command again. If you have :ref:`customised <configuration_customisation>` your docker images, you will have to re-build them prior to running ``quickstart``.
 
-``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Koa to Lilac, run::
+``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Lilac to Maple, run::
 
-    tutor local upgrade --from=koa
+    tutor local upgrade --from=lilac
 
 .. _autocomplete:
 

diff --git a/docs/quickstart.rst b/docs/quickstart.rst
@@ -22,8 +22,8 @@ Yes :) This is what happens when you run ``tutor local quickstart``:
 2. Configuration files are generated from templates.
 3. Docker images are downloaded.
 4. Docker containers are provisioned.
-5. A full, production-ready Open edX platform (`Lilac <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-lilac.master/platform_releases/koa.html>`__ release) is run with docker-compose.
+5. A full, production-ready Open edX platform (`Maple <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-maple.master/platform_releases/maple.html>`__ release) is run with docker-compose.
 
 The whole procedure should require less than 10 minutes, on a server with a good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary.
 
-There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the quickstart installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.
+There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the quickstart installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.
diff --git a/requirements/base.in b/requirements/base.in
@@ -5,3 +5,7 @@ pycryptodome
 jinja2
 kubernetes
 pyyaml>=4.2b1
+
+# Strongly recommended plugins
+tutor-forum>=13.0.0,<14.0.0
+tutor-mfe>=13.0.0,<14.0.0
diff --git a/requirements/plugins.txt b/requirements/plugins.txt
@@ -1,12 +1,12 @@
-# change version ranges when upgrading from lilac
-tutor-android>=12.0.0,<13.0.0
-tutor-discovery>=12.0.0,<13.0.0
-tutor-ecommerce>=12.0.0,<13.0.0
-tutor-forum>=12.0.0,<13.0.0
-tutor-license>=12.0.0,<13.0.0
-tutor-mfe>=12.0.0,<13.0.0
-tutor-minio>=12.0.0,<13.0.0
-tutor-notes>=12.0.0,<13.0.0
-tutor-richie>=12.0.0,<13.0.0
-tutor-webui>=12.0.0,<13.0.0
-tutor-xqueue>=12.0.0,<13.0.0
+# change version ranges when upgrading from maple
+tutor-android>=13.0.0,<14.0.0
+tutor-discovery>=13.0.0,<14.0.0
+tutor-ecommerce>=13.0.0,<14.0.0
+tutor-forum>=13.0.0,<14.0.0
+tutor-license>=13.0.0,<14.0.0
+tutor-mfe>=13.0.0,<14.0.0
+tutor-minio>=13.0.0,<14.0.0
+tutor-notes>=13.0.0,<14.0.0
+tutor-richie>=13.0.0,<14.0.0
+tutor-webui>=13.0.0,<14.0.0
+tutor-xqueue>=13.0.0,<14.0.0
diff --git a/tutor/__about__.py b/tutor/__about__.py
@@ -2,15 +2,15 @@
 
 # Increment this version number to trigger a new release. See
 # docs/tutor.html#versioning for information on the versioning scheme.
-__version__ = "12.2.0"
+__version__ = "13.0.0"
 
 # The version suffix will be appended to the actual version, separated by a
 # dash. Use this suffix to differentiate between the actual released version and
 # the versions from other branches. For instance: set the suffix to "nightly" in
 # the nightly branch.
 # The suffix is cleanly separated from the __version__ in this module to avoid
 # conflicts when merging branches.
-__version_suffix__ = "nightly"
+__version_suffix__ = ""
 
 # The app name will be used to define the name of the default tutor root and
 # plugin directory. To avoid conflicts between multiple locally-installed

diff --git a/tutor/commands/k8s.py b/tutor/commands/k8s.py
@@ -320,7 +320,8 @@ def delete(context: Context, yes: bool) -> None:
 def init(context: Context, limit: Optional[str]) -> None:
     config = tutor_config.load(context.root)
     runner = K8sJobRunner(context.root, config)
-    for name in ["caddy", "elasticsearch", "mysql", "mongodb"]:
+    wait_for_pod_ready(config, "caddy")
+    for name in ["elasticsearch", "mysql", "mongodb"]:
         if tutor_config.is_service_activated(config, name):
             wait_for_pod_ready(config, name)
     jobs.initialise(runner, limit_to=limit)
@@ -442,7 +443,7 @@ def wait(context: Context, name: str) -> None:
     "--from",
     "from_version",
     default="koa",
-    type=click.Choice(["ironwood", "juniper", "koa"]),
+    type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
 )
 @click.pass_obj
 def upgrade(context: Context, from_version: str) -> None:
@@ -461,6 +462,10 @@ def upgrade(context: Context, from_version: str) -> None:
         upgrade_from_koa(config)
         running_version = "lilac"
 
+    if running_version == "lilac":
+        # Nothing to do here
+        running_version = "maple"
+
 
 def upgrade_from_ironwood(config: Config) -> None:
     if not config["RUN_MONGODB"]:

diff --git a/tutor/commands/local.py b/tutor/commands/local.py
@@ -137,6 +137,10 @@ def upgrade(context: click.Context, from_version: str, non_interactive: bool) ->
         upgrade_from_koa(context, config)
         running_version = "lilac"
 
+    if running_version == "lilac":
+        # Nothing to do here
+        running_version = "maple"
+
 
 def upgrade_from_ironwood(context: click.Context, config: Config) -> None:
     click.echo(fmt.title("Upgrading from Ironwood"))

diff --git a/tutor/config.py b/tutor/config.py
@@ -38,7 +38,6 @@ def load_full(root: str) -> Config:
     """
     Load a full configuration, with user, base and defaults.
     """
-    convert_json2yml(root)
     config = get_user(root)
     update_with_base(config)
     update_with_defaults(config)
@@ -84,6 +83,7 @@ def get_user(root: str) -> Config:
 
     Overrides from environment variables are loaded as well.
     """
+    convert_json2yml(root)
     path = config_path(root)
     config = {}
     if os.path.exists(path):

diff --git a/tutor/env.py b/tutor/env.py
@@ -331,9 +331,14 @@ def current_release(root: str) -> str:
     """
     Return the name of the current Open edX release.
     """
-    return {"0": "ironwood", "3": "ironwood", "10": "juniper", "11": "koa"}[
-        current_version(root).split(".")[0]
-    ]
+    return {
+        "0": "ironwood",
+        "3": "ironwood",
+        "10": "juniper",
+        "11": "koa",
+        "12": "lilac",
+        "13": "maple",
+    }[current_version(root).split(".")[0]]
 
 
 def current_version(root: str) -> str:

diff --git a/tutor/templates/apps/openedx/config/cms.env.json b/tutor/templates/apps/openedx/config/cms.env.json
@@ -37,7 +37,7 @@
   "EMAIL_USE_TLS": {{ "true" if SMTP_USE_TLS else "false" }},
   "HTTPS": "{{ "on" if ENABLE_HTTPS else "off" }}",
   "LANGUAGE_CODE": "{{ LANGUAGE_CODE }}",
-  "SESSION_COOKIE_DOMAIN": ".{{ LMS_HOST|common_domain(CMS_HOST) }}",
+  "SESSION_COOKIE_DOMAIN": "{{ CMS_HOST }}",
   {{ patch("cms-env", separator=",\n", suffix=",")|indent(2) }}
   "CACHES": {
     "default": {

diff --git a/tutor/templates/apps/openedx/config/lms.env.json b/tutor/templates/apps/openedx/config/lms.env.json
@@ -45,7 +45,7 @@
   "ACE_ROUTING_KEY": "edx.lms.core.default",
   "HTTPS": "{{ "on" if ENABLE_HTTPS else "off" }}",
   "LANGUAGE_CODE": "{{ LANGUAGE_CODE }}",
-  "SESSION_COOKIE_DOMAIN": ".{{ LMS_HOST|common_domain(CMS_HOST) }}",
+  "SESSION_COOKIE_DOMAIN": "{{ LMS_HOST }}",
   {{ patch("lms-env", separator=",\n", suffix=",")|indent(2) }}
   "CACHES": {
     "default": {