From 10fe660112defcc49116ddbd2812d6732e0f8834 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9gis=20Behmo?= Date: Fri, 15 Apr 2022 10:51:19 +0200 Subject: [PATCH] feat: add --mount/--automount options to local/dev The `--mount` and `--automount` options are available both with `tutor local` and `tutor dev` commands. They allow users to easily bind-mount containers from the host to containers. Yes, I know, we already provide that possibility with the `bindmount` command and the `--volume=/path/` option. But these suffer from the following drawbacks: - They are difficult to understand. - The "bindmount" command name does not make much sense. - It's not convenient to mount an arbitrary folder from the host to multiple containers, such as the many lms/cms containers (web apps, celery workers and job runners). To address this situation, we now recommend to make use of --mount and --automount options. 1. `--mount=service1[,service2,...]:/host/path:/container/path`: manually mount `/host/path` to `/container/path` in container "service1" (and "service2"). 2. `--automount=/host/path`: use the new v1 plugin API to discover plugins that will detect this option and select the right containers in which to bind-mount volumes. This is really nifty... Close https://github.com/overhangio/2u-tutor-adoption/issues/43 --- CHANGELOG.md | 1 + docs/dev.rst | 66 ++++-- tests/commands/test_compose.py | 30 +++ tutor/bindmounts.py | 2 +- tutor/commands/compose.py | 206 ++++++++++++++++-- tutor/commands/dev.py | 18 +- tutor/commands/images.py | 4 +- tutor/commands/local.py | 17 +- tutor/hooks/consts.py | 66 ++++-- tutor/templates/config/defaults.yml | 1 + tutor/templates/dev/docker-compose.jobs.yml | 2 +- tutor/templates/dev/docker-compose.yml | 2 +- tutor/templates/local/docker-compose.jobs.yml | 2 +- tutor/templates/local/docker-compose.prod.yml | 2 +- tutor/templates/local/docker-compose.yml | 2 +- 15 files changed, 354 insertions(+), 67 deletions(-) create mode 100644 tests/commands/test_compose.py diff --git a/CHANGELOG.md b/CHANGELOG.md index aa661461f34..ee7eac6b370 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Note: Breaking changes between versions are indicated by "💥". ## Unreleased +- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev`` commands to auto-magically bind-mount folders from the host. - [Feature] Add `tutor dev quickstart` command, which is similar to `tutor local quickstart`, except that it uses dev containers instead of local production ones and includes some other small differences for the convience of Open edX developers. This should remove some friction from the Open edX development setup process, which previously required that users provision using local producation containers (`tutor local quickstart`) but then stop them and switch to dev containers (`tutor local stop && tutor dev start -d`). - 💥[Improvement] Make it possible to run `tutor k8s exec ` (#636). As a consequence, it is no longer possible to run quoted commands: `tutor k8s exec ""`. Instead, you should remove the quotes: `tutor k8s exec `. - 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS` environment variable. It is now recommended to create a plugin instead. diff --git a/docs/dev.rst b/docs/dev.rst index 283bd4a7d6f..deb76ca077c 100644 --- a/docs/dev.rst +++ b/docs/dev.rst @@ -107,9 +107,43 @@ Sharing directories with containers It may sometimes be convenient to mount container directories on the host, for instance: for editing and debugging. Tutor provides different solutions to this problem. +.. _mount_option: + +Bind-mount volumes with ``--mount`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``run``, ``runserver``, ``init`` and ``start`` subcommands of ``tutor dev`` and ``tutor local`` support the ``-m/--mount`` option (see :option:`tutor dev start -m`) which can take two different forms. The first is explicit:: + + tutor dev start --mount=lms:/path/to/edx-platform:/openedx/edx-platform lms + +And the second is implicit:: + + tutor dev start --mount=/path/to/edx-platform lms + +With the explicit form, the ``--mount`` option means "bind-mount the host folder /path/to/edx-platform to /openedx/edx-platform in the lms container". + +If you use the explicit format, you will quickly realise that you usually want to bind-mount folders in multiple containers at a time. For instance, you will want to bind-mount the edx-platform repository in the "cms" container. To do that, write instead:: + + tutor dev start --mount=lms,cms:/path/to/edx-platform:/openedx/edx-platform lms + +This command line can become cumbersome and inconvenient to work with. But Tutor can be smart about bind-mounting folders to the right containers in the right place when you use the implicit form of the ``--mount`` option. For instance, the following commands are equivalent:: + + # Explicit form + tutor dev start --mount=lms,lms-worker,lms-job,cms,cms-worker,cms-job:/path/to/edx-platform:/openedx/edx-platform lms + # Implicit form + tutor dev start --mount=/path/to/edx-platform lms + +So, when should you *not* be using the implicit form? That would be when Tutor does not know where to bind-mount your host folders. For instance, if you wanted to bind-mount your edx-platform virtual environment located in ``~/venvs/edx-platform``, you should not write ``--mount=~/venvs/edx-platform``, because that folder would be mounted in a way that would override the edx-platform repository in the container. Instead, you should write:: + + tutor dev start --mount=lms:~/venvs/edx-platform:/openedx/venv lms + +.. note:: Remember to setup your edx-platform repository for development! See :ref:`edx_platform_dev_env`. + Bind-mount from the "volumes/" directory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. warning:: Bind-mounting volumes with the ``bindmount`` command is no longer the default, recommended way of bind-mounting volumes from the host. Instead, see the :ref:`mount option `. + Tutor makes it easy to create a bind-mount from an existing container. First, copy the contents of a container directory with the ``bindmount`` command. For instance, to copy the virtual environment of the "lms" container:: tutor dev bindmount lms /openedx/venv @@ -128,6 +162,8 @@ Notice how the ``--volume=/openedx/venv`` option differs from `Docker syntax `. + The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform `__ repository. In such cases, you can simply use the ``-v/--volume`` `Docker option `__:: tutor dev run --volume=/path/to/edx-platform:/openedx/edx-platform lms bash @@ -135,7 +171,7 @@ The above solution may not work for you if you already have an existing director Override docker-compose volumes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The above solutions require that you explicitly pass the ``-v/--volume`` to every ``run`` or ``runserver`` command, which may be inconvenient. Also, these solutions are not compatible with the ``start`` command. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands:: +The above solutions require that you explicitly pass the ``-m/--mount`` options to every ``run``, ``runserver``, ``start`` or ``init`` command, which may be inconvenient. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands:: vim "$(tutor config printroot)/env/dev/docker-compose.override.yml" @@ -156,37 +192,31 @@ You are then free to bind-mount any directory to any container. For instance, to volumes: - /path/to/edx-platform:/openedx/edx-platform -This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v/--volume`` option from the command line with the ``run`` or ``runserver`` commands. +This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-m/--mount`` option from the command line. .. note:: - The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory. + The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory. One-time jobs from initialisation commands load the ``local/docker-compose.jobs.override.yml`` and ``dev/docker-compose.jobs.override.yml``. Common tasks ------------ +.. _edx_platform_dev_env: + Setting up a development environment for edx-platform ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Following the instructions :ref:`above ` on how to bind-mount directories from the host above, you may mount your own `edx-platform `__ fork in your containers by running either:: - - # Mount from the volumes/ directory - tutor dev bindmount lms /openedx/edx-platform - tutor dev runserver --volume=/openedx/edx-platform lms - - # Mount from an arbitrary directory - tutor dev runserver --volume=/path/to/edx-platform:/openedx/edx-platform lms +Following the instructions :ref:`above ` on how to bind-mount directories from the host above, you may mount your own `edx-platform `__ fork in your containers by running:: - # Add your own volumes to $(tutor config printroot)/env/dev/docker-compose.override.yml - tutor dev runserver lms + tutor dev runserver --mount=/path/to/edx-platform lms -If you choose any but the first solution above, you will have to make sure that your fork works with Tutor. +But to achieve that, you will have to make sure that your fork works with Tutor. -First of all, you should make sure that you are working off the ``open-release/maple.2`` tag. See the :ref:`fork edx-platform section ` for more information. +First of all, you should make sure that you are working off the ``open-release/maple.3`` tag. See the :ref:`fork edx-platform section ` for more information. Then, you should run the following commands:: # Run bash in the lms container - tutor dev run [--volume=...] lms bash + tutor dev run --mount=/path/to/edx-platform lms bash # Compile local python requirements pip install --requirement requirements/edx/development.txt @@ -197,9 +227,9 @@ Then, you should run the following commands:: # Rebuild static assets openedx-assets build --env=dev -To debug a local edx-platform repository, add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run:: +After running all these commands, your edx-platform repository will be ready for local development. To debug a local edx-platform repository, you can then add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run:: - tutor dev runserver [--volume=...] lms + tutor dev runserver --mount=/path/to/edx-platform lms XBlock and edx-platform plugin development ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/tests/commands/test_compose.py b/tests/commands/test_compose.py new file mode 100644 index 00000000000..eec5f9b0030 --- /dev/null +++ b/tests/commands/test_compose.py @@ -0,0 +1,30 @@ +import unittest + +from click.exceptions import ClickException +from tutor.commands import compose + + +class ComposeTests(unittest.TestCase): + def test_mount_option_parsing(self) -> None: + param = compose.MountParam() + + self.assertEqual( + [("lms", "/path/to/edx-platform", "/openedx/edx-platform")], + param("lms:/path/to/edx-platform:/openedx/edx-platform"), + ) + self.assertEqual( + [ + ("lms", "/path/to/edx-platform", "/openedx/edx-platform"), + ("cms", "/path/to/edx-platform", "/openedx/edx-platform"), + ], + param("lms,cms:/path/to/edx-platform:/openedx/edx-platform"), + ) + self.assertEqual( + [ + ("lms", "/path/to/edx-platform", "/openedx/edx-platform"), + ("cms", "/path/to/edx-platform", "/openedx/edx-platform"), + ], + param("lms, cms:/path/to/edx-platform:/openedx/edx-platform"), + ) + with self.assertRaises(ClickException): + param("lms,:/path/to/edx-platform:/openedx/edx-platform") diff --git a/tutor/bindmounts.py b/tutor/bindmounts.py index 3f25d6e3a2b..874507dea75 100644 --- a/tutor/bindmounts.py +++ b/tutor/bindmounts.py @@ -36,7 +36,7 @@ def create( "--no-deps", "--user=0", "--volume", - "{}:{}".format(volumes_root_path, container_volumes_root_path), + f"{volumes_root_path}:{container_volumes_root_path}", service, "sh", "-e", diff --git a/tutor/commands/compose.py b/tutor/commands/compose.py index 696b498e897..413d4f55913 100644 --- a/tutor/commands/compose.py +++ b/tutor/commands/compose.py @@ -1,5 +1,6 @@ import os -from typing import List +import re +import typing as t import click @@ -7,22 +8,25 @@ from tutor import config as tutor_config from tutor import env as tutor_env from tutor import fmt, jobs, utils -from tutor.commands.context import BaseJobContext +from tutor import serialize from tutor.exceptions import TutorError from tutor.types import Config +from tutor.commands.context import BaseJobContext +from tutor import hooks class ComposeJobRunner(jobs.BaseComposeJobRunner): def __init__(self, root: str, config: Config): super().__init__(root, config) self.project_name = "" - self.docker_compose_files: List[str] = [] - self.docker_compose_job_files: List[str] = [] + self.docker_compose_files: t.List[str] = [] + self.docker_compose_job_files: t.List[str] = [] def docker_compose(self, *command: str) -> int: """ Run docker-compose with the right yml files. """ + self.__update_docker_compose_tmp() args = [] for docker_compose_path in self.docker_compose_files: if os.path.exists(docker_compose_path): @@ -31,6 +35,35 @@ def docker_compose(self, *command: str) -> int: *args, "--project-name", self.project_name, *command ) + def __update_docker_compose_tmp(self) -> None: + """ + Update the contents of the docker-compose.tmp.yml file, which is generated at runtime. + """ + docker_compose_tmp = { + "version": "{{ DOCKER_COMPOSE_VERSION }}", + "services": {}, + } + docker_compose_jobs_tmp = { + "version": "{{ DOCKER_COMPOSE_VERSION }}", + "services": {}, + } + docker_compose_tmp = hooks.Filters.COMPOSE_LOCAL_TMP.apply(docker_compose_tmp) + docker_compose_jobs_tmp = hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.apply( + docker_compose_jobs_tmp + ) + docker_compose_tmp = tutor_env.render_unknown(self.config, docker_compose_tmp) + docker_compose_jobs_tmp = tutor_env.render_unknown( + self.config, docker_compose_jobs_tmp + ) + tutor_env.write_to( + serialize.dumps(docker_compose_tmp), + tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"), + ) + tutor_env.write_to( + serialize.dumps(docker_compose_jobs_tmp), + tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"), + ) + def run_job(self, service: str, command: str) -> int: """ Run the "{{ service }}-job" service from local/docker-compose.jobs.yml with the @@ -60,16 +93,78 @@ def job_runner(self, config: Config) -> ComposeJobRunner: raise NotImplementedError +class MountParam(click.ParamType): + """ + Parser for --mount arguments of the form "service1[,service2,...]:/host/path:/container/path". + """ + + name = "mount" + MountType = t.Tuple[str, str, str] + # Note that this syntax does not allow us to include colon ':' characters in paths + PARAM_REGEXP = ( + r"(?P[a-zA-Z0-9_, ]+):(?P[^:]+):(?P[^:]+)" + ) + + def convert( + self, + value: str, + param: t.Optional["click.Parameter"], + ctx: t.Optional[click.Context], + ) -> t.List["MountType"]: + mounts: t.List["MountParam.MountType"] = [] + match = re.match(self.PARAM_REGEXP, value) + if match: + # Argument is of the form "containers:/host/path:/container/path" + services: t.List[str] = [ + service.strip() for service in match["services"].split(",") + ] + host_path = os.path.abspath(os.path.expanduser(match["host_path"])) + host_path = host_path.replace(os.path.sep, "/") + container_path = match["container_path"] + for service in services: + if not service: + self.fail( + f"incorrect services syntax: '{match['services']}'", param, ctx + ) + mounts.append((service, host_path, container_path)) + else: + # Argument is of the form "/host/path" + host_path = os.path.abspath(os.path.expanduser(value)) + volumes: t.Iterator[ + t.Tuple[str, str] + ] = hooks.Filters.COMPOSE_MOUNTS.iterate(os.path.basename(host_path)) + for service, container_path in volumes: + mounts.append((service, host_path, container_path)) + if not mounts: + raise self.fail(f"no mount found for {value}", param, ctx) + return mounts + + +mount_option = click.option( + "-m", + "--mount", + "mounts", + help="""Bind-mount a folder from the host in the right containers. This option can take two different forms. The first one is explicit: 'service1[,service2...]:/host/path:/container/path'. The other is implicit: '/host/path'. Arguments passed in the implicit form will be parsed by plugins to define the right folders to bind-mount from the host.""", + type=MountParam(), + multiple=True, +) + + @click.command( short_help="Run all or a selection of services.", help="Run all or a selection of services. Docker images will be rebuilt where necessary.", ) @click.option("--skip-build", is_flag=True, help="Skip image building") @click.option("-d", "--detach", is_flag=True, help="Start in daemon mode") +@mount_option @click.argument("services", metavar="service", nargs=-1) @click.pass_obj def start( - context: BaseComposeContext, skip_build: bool, detach: bool, services: List[str] + context: BaseComposeContext, + skip_build: bool, + detach: bool, + mounts: t.Tuple[t.List[MountParam.MountType]], + services: t.List[str], ) -> None: command = ["up", "--remove-orphans"] if not skip_build: @@ -77,6 +172,8 @@ def start( if detach: command.append("-d") + process_mount_arguments(mounts) + # Start services config = tutor_config.load(context.root) context.job_runner(config).docker_compose(*command, *services) @@ -85,7 +182,7 @@ def start( @click.command(help="Stop a running platform") @click.argument("services", metavar="service", nargs=-1) @click.pass_obj -def stop(context: BaseComposeContext, services: List[str]) -> None: +def stop(context: BaseComposeContext, services: t.List[str]) -> None: config = tutor_config.load(context.root) context.job_runner(config).docker_compose("stop", *services) @@ -97,7 +194,7 @@ def stop(context: BaseComposeContext, services: List[str]) -> None: @click.option("-d", "--detach", is_flag=True, help="Start in daemon mode") @click.argument("services", metavar="service", nargs=-1) @click.pass_context -def reboot(context: click.Context, detach: bool, services: List[str]) -> None: +def reboot(context: click.Context, detach: bool, services: t.List[str]) -> None: context.invoke(stop, services=services) context.invoke(start, detach=detach, services=services) @@ -111,7 +208,7 @@ def reboot(context: click.Context, detach: bool, services: List[str]) -> None: ) @click.argument("services", metavar="service", nargs=-1) @click.pass_obj -def restart(context: BaseComposeContext, services: List[str]) -> None: +def restart(context: BaseComposeContext, services: t.List[str]) -> None: config = tutor_config.load(context.root) command = ["restart"] if "all" in services: @@ -130,8 +227,14 @@ def restart(context: BaseComposeContext, services: List[str]) -> None: @click.command(help="Initialise all applications") @click.option("-l", "--limit", help="Limit initialisation to this service or plugin") +@mount_option @click.pass_obj -def init(context: BaseComposeContext, limit: str) -> None: +def init( + context: BaseComposeContext, + limit: str, + mounts: t.Tuple[t.List[MountParam.MountType]], +) -> None: + process_mount_arguments(mounts) config = tutor_config.load(context.root) runner = context.job_runner(config) jobs.initialise(runner, limit_to=limit) @@ -177,7 +280,9 @@ def createuser( ) @click.argument("theme_name") @click.pass_obj -def settheme(context: BaseComposeContext, domains: List[str], theme_name: str) -> None: +def settheme( + context: BaseComposeContext, domains: t.List[str], theme_name: str +) -> None: config = tutor_config.load(context.root) runner = context.job_runner(config) domains = domains or jobs.get_all_openedx_domains(config) @@ -202,9 +307,15 @@ def importdemocourse(context: BaseComposeContext) -> None: ), context_settings={"ignore_unknown_options": True}, ) +@mount_option @click.argument("args", nargs=-1, required=True) @click.pass_context -def run(context: click.Context, args: List[str]) -> None: +def run( + context: click.Context, + mounts: t.Tuple[t.List[MountParam.MountType]], + args: t.List[str], +) -> None: + process_mount_arguments(mounts) extra_args = ["--rm"] if not utils.is_a_tty(): extra_args.append("-T") @@ -221,6 +332,9 @@ def run(context: click.Context, args: List[str]) -> None: @click.argument("path") @click.pass_obj def bindmount_command(context: BaseComposeContext, service: str, path: str) -> None: + """ + This command is made obsolete by the --mount arguments. + """ config = tutor_config.load(context.root) host_path = bindmounts.create(context.job_runner(config), service, path) fmt.echo_info( @@ -241,7 +355,7 @@ def bindmount_command(context: BaseComposeContext, service: str, path: str) -> N ) @click.argument("args", nargs=-1, required=True) @click.pass_context -def execute(context: click.Context, args: List[str]) -> None: +def execute(context: click.Context, args: t.List[str]) -> None: context.invoke(dc_command, command="exec", args=args) @@ -281,7 +395,7 @@ def status(context: click.Context) -> None: @click.argument("command") @click.argument("args", nargs=-1) @click.pass_obj -def dc_command(context: BaseComposeContext, command: str, args: List[str]) -> None: +def dc_command(context: BaseComposeContext, command: str, args: t.List[str]) -> None: config = tutor_config.load(context.root) volumes, non_volume_args = bindmounts.parse_volumes(args) volume_args = [] @@ -299,6 +413,72 @@ def dc_command(context: BaseComposeContext, command: str, args: List[str]) -> No context.job_runner(config).docker_compose(command, *volume_args, *non_volume_args) +def process_mount_arguments(mounts: t.Tuple[t.List[MountParam.MountType]]) -> None: + """ + Process --mount arguments. + + Most docker-compose commands support --mount arguments. This option + is used to bind-mount folders from the host. A docker-compose.tmp.yml is + generated at runtime and includes the bind-mounted volumes that were passed as CLI + arguments. + + Bind-mounts that are associated to "*-job" services will be added to the + docker-compose jobs file. + """ + app_mounts: t.List[MountParam.MountType] = [] + job_mounts: t.List[MountParam.MountType] = [] + for mount in mounts: + for service, host_path, container_path in mount: + if service.endswith("-job"): + job_mounts.append((service, host_path, container_path)) + else: + app_mounts.append((service, host_path, container_path)) + + def _add_mounts( + docker_compose: t.Dict[str, t.Any], bind_mounts: t.List[MountParam.MountType] + ) -> t.Dict[str, t.Any]: + services = docker_compose.setdefault("services", {}) + for service, host_path, container_path in bind_mounts: + fmt.echo_info(f"Bind-mount: {host_path} -> {container_path} in {service}") + services.setdefault(service, {"volumes": []}) + services[service]["volumes"].append(f"{host_path}:{container_path}") + return docker_compose + + # Save bind-mounts + @hooks.Filters.COMPOSE_LOCAL_TMP.add() + def _add_mounts_to_docker_compose_tmp( + docker_compose_tmp: t.Dict[str, t.Any] + ) -> t.Dict[str, t.Any]: + return _add_mounts(docker_compose_tmp, app_mounts) + + @hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.add() + def _add_mounts_to_docker_compose_jobs_tmp( + docker_compose_tmp: t.Dict[str, t.Any] + ) -> t.Dict[str, t.Any]: + return _add_mounts(docker_compose_tmp, job_mounts) + + +@hooks.Filters.COMPOSE_MOUNTS.add() +def _mount_edx_platform( + volumes: t.List[t.Tuple[str, str]], name: str +) -> t.List[t.Tuple[str, str]]: + """ + When mounting edx-platform with `--mount=/path/to/edx-platform`, bind-mount the host + repo in the lms/cms containers. + """ + if name == "edx-platform": + path = "/openedx/edx-platform" + volumes += [ + ("lms", path), + ("cms", path), + ("lms-worker", path), + ("cms-worker", path), + ("lms-job", path), + ("cms-job", path), + ] + return volumes + + def add_commands(command_group: click.Group) -> None: command_group.add_command(start) command_group.add_command(stop) diff --git a/tutor/commands/dev.py b/tutor/commands/dev.py index c8a647ea077..9bfe54255b6 100644 --- a/tutor/commands/dev.py +++ b/tutor/commands/dev.py @@ -1,4 +1,4 @@ -from typing import List +import typing as t import click @@ -21,12 +21,14 @@ def __init__(self, root: str, config: Config): self.docker_compose_files += [ tutor_env.pathjoin(self.root, "local", "docker-compose.yml"), tutor_env.pathjoin(self.root, "dev", "docker-compose.yml"), + tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"), tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"), tutor_env.pathjoin(self.root, "dev", "docker-compose.override.yml"), ] self.docker_compose_job_files += [ tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"), tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.yml"), + tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"), tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"), tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.override.yml"), ] @@ -99,21 +101,25 @@ def quickstart(context: click.Context, non_interactive: bool, pullimages: bool) help="Run a development server", context_settings={"ignore_unknown_options": True}, ) +@compose.mount_option @click.argument("options", nargs=-1, required=False) @click.argument("service") @click.pass_context -def runserver(context: click.Context, options: List[str], service: str) -> None: +def runserver( + context: click.Context, + mounts: t.Tuple[t.List[compose.MountParam.MountType]], + options: t.List[str], + service: str, +) -> None: config = tutor_config.load(context.obj.root) if service in ["lms", "cms"]: port = 8000 if service == "lms" else 8001 host = config["LMS_HOST"] if service == "lms" else config["CMS_HOST"] fmt.echo_info( - "The {} service will be available at http://{}:{}".format( - service, host, port - ) + f"The {service} service will be available at http://{host}:{port}" ) args = ["--service-ports", *options, service] - context.invoke(compose.run, args=args) + context.invoke(compose.run, mounts=mounts, args=args) dev.add_command(quickstart) diff --git a/tutor/commands/images.py b/tutor/commands/images.py index c9f43adf4d3..a069d447437 100644 --- a/tutor/commands/images.py +++ b/tutor/commands/images.py @@ -174,7 +174,7 @@ def find_images_to_build( ] = hooks.Filters.IMAGES_BUILD.iterate(config) found = False for name, path, tag, args in all_images_to_build: - if name == image or image == "all": + if image in [name, "all"]: found = True tag = tutor_env.render_str(config, tag) yield (name, path, tag, args) @@ -196,7 +196,7 @@ def find_remote_image_tags( all_remote_images: t.Iterator[t.Tuple[str, str]] = filtre.iterate(config) found = False for name, tag in all_remote_images: - if name == image or image == "all": + if image in [name, "all"]: found = True yield tutor_env.render_str(config, tag) if not found: diff --git a/tutor/commands/local.py b/tutor/commands/local.py index c9bd8fc4147..8b95256e5eb 100644 --- a/tutor/commands/local.py +++ b/tutor/commands/local.py @@ -1,4 +1,4 @@ -from typing import Optional +import typing as t import click @@ -22,10 +22,12 @@ def __init__(self, root: str, config: Config): self.docker_compose_files += [ tutor_env.pathjoin(self.root, "local", "docker-compose.yml"), tutor_env.pathjoin(self.root, "local", "docker-compose.prod.yml"), + tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"), tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"), ] self.docker_compose_job_files += [ tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"), + tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"), tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"), ] @@ -46,7 +48,12 @@ def local(context: click.Context) -> None: @click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively") @click.option("-p", "--pullimages", is_flag=True, help="Update docker images") @click.pass_context -def quickstart(context: click.Context, non_interactive: bool, pullimages: bool) -> None: +def quickstart( + context: click.Context, + mounts: t.Tuple[t.List[compose.MountParam.MountType]], + non_interactive: bool, + pullimages: bool, +) -> None: try: utils.check_macos_docker_memory() except exceptions.TutorError as e: @@ -113,9 +120,9 @@ def quickstart(context: click.Context, non_interactive: bool, pullimages: bool) click.echo(fmt.title("Docker image updates")) context.invoke(compose.dc_command, command="pull") click.echo(fmt.title("Starting the platform in detached mode")) - context.invoke(compose.start, detach=True) + context.invoke(compose.start, mounts=mounts, detach=True) click.echo(fmt.title("Database creation and migrations")) - context.invoke(compose.init) + context.invoke(compose.init, mounts=mounts) config = tutor_config.load(context.obj.root) fmt.echo_info( @@ -142,7 +149,7 @@ def quickstart(context: click.Context, non_interactive: bool, pullimages: bool) type=click.Choice(["ironwood", "juniper", "koa", "lilac"]), ) @click.pass_context -def upgrade(context: click.Context, from_release: Optional[str]) -> None: +def upgrade(context: click.Context, from_release: t.Optional[str]) -> None: fmt.echo_alert( "This command only performs a partial upgrade of your Open edX platform. " "To perform a full upgrade, you should run `tutor local quickstart`." diff --git a/tutor/hooks/consts.py b/tutor/hooks/consts.py index 3b9e1528140..e53a155deed 100644 --- a/tutor/hooks/consts.py +++ b/tutor/hooks/consts.py @@ -97,6 +97,55 @@ def your_filter(items): return items """ + #: List of commands to be executed during initialization. These commands typically + #: include database migrations, setting feature flags, etc. + #: + #: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. + #: + #: - ``service`` is the name of the container in which the task will be executed. + #: - ``path`` is a tuple that corresponds to a template relative path. + #: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`). + #: The command to execute will be read from that template, after it is rendered. + COMMANDS_INIT = filters.get("commands:init") + + #: List of commands to be executed prior to initialization. These commands are run even + #: before the mysql databases are created and the migrations are applied. + #: + #: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. (see :py:data:`COMMANDS_INIT`). + COMMANDS_PRE_INIT = filters.get("commands:pre-init") + + #: List of folders to bind-mount in docker-compose containers, either in ``tutor local`` or ``tutor dev``. + #: + #: Many ``tutor local`` and ``tutor dev`` commands support ``--mounts`` options + #: that allow plugins to define custom behaviour at runtime. For instance + #: ``--mount=/path/to/edx-platform`` would cause this host folder to be + #: bind-mounted in different containers (lms, lms-worker, cms, cms-worker) at the + #: /openedx/edx-platform location. Plugin developers may implement this filter to + #: define custom behaviour when mounting folders that relate to their plugins. For + #: instance, the ecommerce plugin may process the ``--mount=/path/to/ecommerce`` + #: option. + #: + #: :parameter list[tuple[str, str]] mounts: each item is a ``(service, path)`` + #: tuple, where ``service`` is the name of the docker-compose service and ``path`` is + #: the location in the container where the folder should be bind-mounted. Note: the + #: path must be slash-separated ("/"). Thus, do not use ``os.path.join`` to generate + #: the ``path`` because it will fail on Windows. + #: :parameter str name: basename of the host-mounted folder. In the example above, + #: this is "edx-platform". When implementing this filter you should check this name to + #: conditionnally add mounts. + COMPOSE_MOUNTS = filters.get("compose:mounts") + + #: Contents of the local/docker-compose.tmp.yml file that will be generated at + #: runtime. This is used for instance to bind-mount folders from the host (see + #: :py:data:`COMPOSE_MOUNTS`) + #: + #: :parameter dict[str, ...] docker_compose_tmp: values which will be serialized to local/docker-compose.tmp.yml. + #: Keys and values will be rendered before saving, such that you may include ``{{ ... }}`` statements. + COMPOSE_LOCAL_TMP = filters.get("compose:local:tmp") + + #: Same as :py:data:`COMPOSE_LOCAL_TMP` but for jobs + COMPOSE_LOCAL_JOBS_TMP = filters.get("compose:local-jobs:tmp") + #: List of images to be built when we run ``tutor images build ...``. #: #: :parameter list[tuple[str, tuple[str, ...], str, tuple[str, ...]]] tasks: list of ``(name, path, tag, args)`` tuples. @@ -125,23 +174,6 @@ def your_filter(items): #: Parameters are the same as for :py:data:`IMAGES_PULL`. IMAGES_PUSH = filters.get("images:push") - #: List of commands to be executed during initialization. These commands typically - #: include database migrations, setting feature flags, etc. - #: - #: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. - #: - #: - ``service`` is the name of the container in which the task will be executed. - #: - ``path`` is a tuple that corresponds to a template relative path. - #: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`). - #: The command to execute will be read from that template, after it is rendered. - COMMANDS_INIT = filters.get("commands:init") - - #: List of commands to be executed prior to initialization. These commands are run even - #: before the mysql databases are created and the migrations are applied. - #: - #: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. (see :py:data:`COMMANDS_INIT`). - COMMANDS_PRE_INIT = filters.get("commands:pre-init") - #: List of command line interface (CLI) commands. #: #: :parameter list commands: commands are instances of ``click.Command``. They will diff --git a/tutor/templates/config/defaults.yml b/tutor/templates/config/defaults.yml index db2f67803e0..1628333a4a2 100644 --- a/tutor/templates/config/defaults.yml +++ b/tutor/templates/config/defaults.yml @@ -8,6 +8,7 @@ CMS_OAUTH2_KEY_SSO: "cms-sso" CMS_OAUTH2_KEY_SSO_DEV: "cms-sso-dev" CONTACT_EMAIL: "contact@{{ LMS_HOST }}" DEV_PROJECT_NAME: "{{ TUTOR_APP }}_dev" +DOCKER_COMPOSE_VERSION: "3.7" DOCKER_REGISTRY: "docker.io/" DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION }}" DOCKER_IMAGE_OPENEDX_DEV: "openedx-dev" diff --git a/tutor/templates/dev/docker-compose.jobs.yml b/tutor/templates/dev/docker-compose.jobs.yml index 3e50ba37b65..9dfa7c68fc2 100644 --- a/tutor/templates/dev/docker-compose.jobs.yml +++ b/tutor/templates/dev/docker-compose.jobs.yml @@ -1,3 +1,3 @@ -version: "3.7" +version: "{{ DOCKER_COMPOSE_VERSION }}" services: {% if not patch("dev-docker-compose-jobs-services") %}{}{% endif %} {{ patch("dev-docker-compose-jobs-services")|indent(4) }} diff --git a/tutor/templates/dev/docker-compose.yml b/tutor/templates/dev/docker-compose.yml index 06e85bcc023..784a97e6103 100644 --- a/tutor/templates/dev/docker-compose.yml +++ b/tutor/templates/dev/docker-compose.yml @@ -1,4 +1,4 @@ -version: "3.7" +version: "{{ DOCKER_COMPOSE_VERSION }}" x-openedx-service: &openedx-service diff --git a/tutor/templates/local/docker-compose.jobs.yml b/tutor/templates/local/docker-compose.jobs.yml index cf37413042a..ed7f967f44c 100644 --- a/tutor/templates/local/docker-compose.jobs.yml +++ b/tutor/templates/local/docker-compose.jobs.yml @@ -1,4 +1,4 @@ -version: "3.7" +version: "{{ DOCKER_COMPOSE_VERSION }}" services: mysql-job: diff --git a/tutor/templates/local/docker-compose.prod.yml b/tutor/templates/local/docker-compose.prod.yml index b753bfa847b..b803b41afb6 100644 --- a/tutor/templates/local/docker-compose.prod.yml +++ b/tutor/templates/local/docker-compose.prod.yml @@ -1,4 +1,4 @@ -version: "3.7" +version: "{{ DOCKER_COMPOSE_VERSION }}" services: # Web proxy for load balancing and SSL termination caddy: diff --git a/tutor/templates/local/docker-compose.yml b/tutor/templates/local/docker-compose.yml index adc92626564..940d41fe90c 100644 --- a/tutor/templates/local/docker-compose.yml +++ b/tutor/templates/local/docker-compose.yml @@ -1,4 +1,4 @@ -version: "3.7" +version: "{{ DOCKER_COMPOSE_VERSION }}" services: ############# External services