Tutor Adoption 4: Devstack Parity: Improve Dev Workflows

[kdmccormick]

Background

#144 describes three options for improving tutor dev workflows around locally-changed code and requirements. This is one option.

### Tasks
- [x] https://github.com/openedx/wg-developer-experience/issues/152
- [x] https://github.com/overhangio/tutor/pull/813
- [ ] https://github.com/openedx/edx-platform/issues/31798
- [ ] https://github.com/openedx/wg-developer-experience/issues/150
- [ ] https://github.com/openedx/wg-devops/issues/22
- [ ] https://github.com/openedx/wg-developer-experience/issues/162
- [ ] https://github.com/openedx/wg-devops/issues/21
- [ ] https://github.com/kdmccormick/tutor/pull/32
- [ ] https://github.com/kdmccormick/tutor/pull/33
- [ ] https://github.com/openedx/wg-developer-experience/issues/177
- [ ] https://github.com/openedx/wg-developer-experience/issues/159

Notes

These tasks are based on this comment from the quickdev TEP:

I too want to make sure we are solving the right problems rather than applying workarounds to things that are broken on a deeper level. Let me break down the problems I see with tutor dev as a it stands.

node_modules

I’m no npm expert, but from my research it seems that npm really wants node_modules to be stored in the root of the repository itself. The ways around this I can think of are:

1. Install all packages in “global” mode (npm -g) so that they are installed somewhere in the container outside of the repository. In a prototype I was able to hackily achieve this by shadowing the npm binary with a script that forwarded all arguments to npm -g. This would also require further unraveling of the paver-based assets pipeline, which is a good thing anyway, and is something I think we could get 2U’s help on.
2. In the dev Docker image, move /openedx/edx-platform/node_modules to /openedx/node_modules and replace the former with a link to the latter. Then, do the same thing for mounted repositories using an ENTRYPOINT script. I had this idea working in an earlier version of quickdev, but dropped it because it seemed too hacky.
3. For micro-frontends, stop trying to use Docker containers in development mode. For backend services, just accept that we will have this problem until all legacy frontends are removed and our backends are purely JSON API servers.

python virtualenv

Thankfully, since the venv is stored at /openedx/venv, which is outside of edx-platform, running pip install -r requirements/dev.txt isn’t required after mounting a repository. However, people often want to modify Python requirements and/or install local versions of packages without too much of a hassle.
Outside of turning the venv into a named volume, I think the best improvement we could make would be to speed up the dev image build as much as possible. That way, it is much faster for developers to update edx-platform’s requirements lists and re-build the image. A few ideas I have:

• Convert GitHub dependencies to PyPI requirements wherever possible. This is already in progress, and I could push to expedite if we decide against quickdev.
• Hunt down and remove extraneous edx-platform requirements. Easier said than done, but it’s something the project ought to do at some point anyway.
• Convince Docker to use the downloaded & cached version of the openedx image when re-building openedx-dev. That is: for me, at least, it seems like Docker wants to re-build the image from scratch when I run tutor dev dc build lms, even if I just ran tutor images pull openedx… there’s got do be something we can do to re-use those downloaded stages from the production image.

static assets

I’m not very knowledgable here but I’m sure the situation could be improved upstream. There’s discussion about the asset pipeline on this github issue. Even without overhauling the whole edx-platform asset pipeline, I have to hope that’s there’s something we could do to move the generated assets out of edx-platform…

jobs

This one’s already in progress but it’s worth mentioning anyway: having the pluggable jobs framework merged should help us condense any remaining “mounting repository preparation” commands down into single command, e.g. tutor dev do prepare-mounted-platform --mount=edx-platform.

egg-info

This one’s fairly minor, but as far as I can tell, edx-platform will always need Open_edX.egg-info file to be present in the repository. If we don’t use a named volume, this means that pip install -e . is non-optional. I think the jobs framework, mentioned above, could handle this nicely.

[regisb]

I think it's great that you are attempting to resolve the underlying issues by attacking edx-platform directly. It's certainly the most difficult path forward, but also the one with the most potential. Let me know how if I can help.

[bradenmacdonald]

python virtualenv

For me one of the recurring problems is that the edx-platform dependencies change all the time...

$ tutor dev start
... gives some error ...
$ tutor quickdev pip-restore
$ tutor dev start
... now it works

But later I need a custom edx-platform:

tutor dev start -m ~/edx-platform
... gives some error depending on what's changed, like: ImportError: cannot import name 'VerticalBlockRenderCompleted' from 'openedx_filters.learning.filters'
$ tutor dev run -m ~/edx-platform lms pip install -r requirements/edx/development.txt
$ tutor dev run -m ~/edx-platform lms ./manage.py lms migrate 
... now it works

What I think we need is some way that we can automatically detect when there's a difference between the requirements specified in the current requirements/*.txt file(s) and the actual venv in use, and then automatically update the venv to match before launching the dev environment. If this were possible, it would solve the primary annoyance I have using Tutor as a devstack, which is that it's very unclear what pieces are working together and which command will "reset" things so that the right tutor version is used with the right edx-platform version is used with the right venv is updated to the right requirements versions so that everything works.

Or another way of phrasing that is, can things like #153 and openedx/edx-platform#31505 take us far enough that we run pip install automatically every time we start tutor dev, so that the "right" requirements are always installed in the venv ?

[kdmccormick]

Sorry Braden, I read your comment last month and totally forgot to respond. The workflow you bring up is a very common one and it's something I've been thinking about a lot!

Something that makes this issue more challenging/interesting is that there are a bunch of containers all running edx-platform: lms, cms, their -worker variants, potentially their -job variants, and potentially any number of ephemeral run containers. Each container has its own /openedx/venv, and the changes to the venv are destroyed when the container is destroyed. So, in order to ensure updated requirements at runtime we'd need to run pip install within each container upon startup. Out of curiosity, I tried this, and it went about as poorly as you'd expect :P

Quickdev addresses this by using a shared, persisted volume for /openedx/venv across all containers. In that world, I bet it'd be possible to run pip install once whenever the platform is started. I believe fixes like #153 and openedx/edx-platform#31505 would bring a no-op pip install down to the realm of 10-30 seconds, making this strategy fairly reasonable.

Back to Tutor proper, though: without a shared /openedx/venv, updating requirements at runtime just does not seem viable. Instead, I think we'd need to make use of the Docker build, which is designed for the use case "when something changes, rebuild the affected parts (and only the affected parts)". Specifically, I can see two way of leveraging the Docker build:

1. When edx-platform is mounted, copy requirements/edx/development.txt (and package-lock.json, while we're at it) into the build context. Then, in the Dockerfile, use those requirements lists as input to the openedx-dev image. That would trigger a requirements rebuild whenever starting or rebooting a platform with modified requirements.
2. Have edx-platform developers work within the build context. That is: ask developers to move their repo to $(tutor config printroot)/env/build/openedx/edx-platform. Then, in the Dockerfile, we could use the repo as input to the openedx-dev image. Seems crazy at first, but it might get us a lot: all this stuff around requirement management, static asset building, and running setup.py could be handled at the Dockerfile level instead of a hand-rolled initialization system.

Sorry, I know that's lot, but I would love to hear your thoughts when you have the chance @bradenmacdonald and @regisb . At the moment, requirements management for tutor dev is noticably harder than it is in Devstack. With some smart refactoring, though, I think we could make it noticably easier and more reliable than it is in Devstack.

[bradenmacdonald]

@kdmccormick Thanks! I'm not knowledgeable enough to predict how well your two ideas would work, but they're great ideas and I'd be very interested in seeing what happens if we try them out.

[regisb]

Option 2 seems quite inconvenient. In my experience, developers are not keen on moving their repos. But I might be wrong.

I need to brainstorm this a little. FTR improving the dev workflow is also a top priority for me, and I agree that it's just a little too hard to hack on edx-platform as it is.

Would it make our lives easier if we moved /openedx/venv to /openedx/edx-platform/.venv? That way, an edx-platform's venv would be automatically bind-mounted every time we bind-mount edx-platform. I realize it would be a sort of step backward, as we are trying to move stuff away from edx-platform... Still, it's probably worth considering.

[kdmccormick]

Would it make our lives easier if we moved /openedx/venv to /openedx/edx-platform/.venv? That way, an edx-platform's venv would be automatically bind-mounted every time we bind-mount edx-platform. I realize it would be a sort of step backward, as we are trying to move stuff away from edx-platform... Still, it's probably worth considering.

Agreed, it is worth considering. It makes me realize that bind-mounted tutor dev is currently in an awkward middle ground. One very important thing lives in the image:

• the Python venv

whereas the rest of it lives in the bind-mount:

• the Python requirements list
• package-lock.json
• node_modules
• compiled assets

Perhaps we should pick a side. I am interesting in prototyping the first one. If you prototype the second one, we could compare results.

Install/compile everything into the image

This is the direction I've been heading. Firstly, I would do either (1) or (2) from my previous comment.

Then, I would implement #150 and openedx/wg-devops#22, or some variation of those issues.

Install/compile everything into the bind-mount

We would move /openedx/venv to /openedx/edx-platform/.venv. We could pip-install Python requirements again when initializating a bind-mounted platform. This would probably work well enough for Linux.

Unfortunately, writing to bind-mounts on macOS is catastrophically slow, so asking those devs to run pip install (or, for that matter, npm install) into the bind-mount is just cruel. However, we could copy the node_modules and .venv directories down from the image to the host.

The current implementation of the copyfrom command, unfortunately, will not work well on macOS, because it's implemented using a bind-mount. I think it could be reimplemented using docker cp, though.

[bradenmacdonald]

Unfortunately, writing to bind-mounts on macOS is catastrophically slow, so asking those devs to run pip install (or, for that matter, npm install) into the bind-mount is just cruel. However, we could copy the node_modules and .venv directories down from the image to the host.

FWIW I measured this just now; once the pip cache is populated, pip install -r requirements/edx/base.txt takes 1m5s using a Docker volume but 4m15s using a bind mount (virtiofs). I imagine the platform would also run slower if python has to read all of the .pyc files out of a bind mount instead of just some.

[regisb]

Let's try to consider this issue from the perspective of a macOS user, as it appears that it's the environment with the most constraints. What is the "right" way to run a containerized development environment in this context? Is there any hope at all, outside of running a native ./manage.py lms runserver on the host?

[kdmccormick]

The major macOS constraint I've become aware of is don't write anything significant to bind-mounts. The two solutions that leads me to are:

• Use named volumes instead of bindmounts (ie, quickdev). This works well, but it introduces spooky background state into everyone's development environment.
• Build as much as possible into the image (options 1 and 2 above). I haven't tried this yet, but I'm eager to.

FWIW, I do have a Mac to test on now.

[bradenmacdonald]

Yeah, using a virtual environment in the container is totally fine, as long as it's not bind-mounted. The problem is just that python doesn't have a simple/fast way to keep that venv in sync with the requirements files. Ideally there would be some readonly venv in the base image, and a writeable venv layer over top of it, and something that automatically rebuilds the writable venv delta whenever it no longer matches the requirements files. This could be as simple as a hacky script to make a hash of the requirements files and write that to the venv folder after each pip install, plus repeating that automatically when the hash doesn't match.

[regisb]

I am tempted to say that optimizing bind-mounts for macOS should not be our job, but then it seems that there are no practical solutions proposed upstream: docker/for-mac#3677

So we don't really have a choice, right? I am still not in favour of named volumes, so the only option left is to build as much as possible into the image, such that the bind-mounted volumes remain as small as possible.

The good news is that I think we can achieve this without burdening the developers too much. I have a working proof of concept that works as follows:

• At build-time, bind-mount the source repo in the image using a build-context. See example 3 from this article and this section of the Docker docs. Thanks to the --build-context option we should be able to implement a tutor images build --bind-mount=/path/to/edx-platform option.
• Aggressively cache pip install and npm clean-install commands using RUN --mount=type=cache,... commands. In Olive my proof of concept is still slowed-down by edx-platform editable requirements but this will get better in Palm thanks to Kyle's efforts.
• Persistent bind-mounts which allow developers to run tutor dev bind-mount /path/to/edx-platform once instead of adding the -m /path/to/edx-platform to every command. I haven't implemented this command yet, but it should not be too hard.
• Move /openedx/edx-platform/node_modules to /openedx/node_modules and symlink to it (Kyle this is your option 2 above). I have just attempted to do this and it seems to be working as expected.

So I think we can attempt to resolve this issue using this approach.

Ideally there would be some readonly venv in the base image, and a writeable venv layer over top of it

This is a really interesting idea. I understand we could achieve that goal with a quick-and-dirty script, but I wonder if there is any "official" way to run a combination of multiple virtualenv?

[bradenmacdonald]

This is a really interesting idea. I understand we could achieve that goal with a quick-and-dirty script, but I wonder if there is any "official" way to run a combination of multiple virtualenv?

Doesn't docker do this for us already though, if the base venv layer is established during the build, then it becomes readonly and any changes made it the container will be done in a copy-on-write overlay of the venv, as long as it's not bind mounted.

[kdmccormick]

We've made a lot of progress here, so I'm closing this, with two remaining follow-up issues:

• Optimize Tutor's openedx image openedx/wg-devops#21
• Move artifacts outside of edx-platform in Tutor's openedx image openedx/wg-devops#22