Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize Tutor's openedx image #21

Open
5 of 10 tasks
Tracked by #146
kdmccormick opened this issue Mar 19, 2023 · 0 comments
Open
5 of 10 tasks
Tracked by #146

Optimize Tutor's openedx image #21

kdmccormick opened this issue Mar 19, 2023 · 0 comments
Assignees
Labels
tutor Requires a change to Tutor

Comments

@kdmccormick
Copy link
Member

kdmccormick commented Mar 19, 2023

Background

See parent issue

Tasks

Notes

No response

@kdmccormick kdmccormick self-assigned this Mar 19, 2023
@kdmccormick kdmccormick changed the title Speed up dev image build Speed up Tutor image builds for developers Mar 19, 2023
@kdmccormick kdmccormick added the epic Large unit of work, consisting of multiple tasks label Mar 19, 2023
regisb referenced this issue in overhangio/tutor Apr 28, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Apr 28, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Apr 28, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Apr 28, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Apr 28, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor May 2, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor May 4, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor May 9, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor May 10, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor May 11, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor May 22, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor May 23, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Jun 5, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Jun 5, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Jun 5, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Jun 5, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Jun 7, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Jun 13, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
regisb referenced this issue in overhangio/tutor Jun 14, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Jul 25, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal.
* Deprecate patches that no longer exist or have changed.
* Announce that buildkit is required (or pull that into a separate PR).

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Jul 25, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal.
* Deprecate patches that no longer exist or have changed.
* Announce that buildkit is required (or pull that into a separate PR).

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 2, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal.
* Deprecate patches that no longer exist or have changed.
* Announce that buildkit is required (or pull that into a separate PR).

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 14, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal.
* Deprecate patches that no longer exist or have changed.
* Announce that buildkit is required (or pull that into a separate PR).

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 18, 2023
TODOs:
* Split buildkit assumption into a separate commit.
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal.
* Deprecate patches that no longer exist or have changed.
* Announce that buildkit is required (or pull that into a separate PR).

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 18, 2023
TODOs:
* Split buildkit assumption into a separate commit.
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal.
* Deprecate patches that no longer exist or have changed.
* Announce that buildkit is required (or pull that into a separate PR).

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 18, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal?
* Deprecate patches that no longer exist or have changed.

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 31, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal?
* Deprecate patches that no longer exist or have changed.

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/edx-platform Aug 31, 2023
Sass compilation includes the LMS's "vendored-in" CSS libraries at
lms/static/css/vendor as a source, and output CSS is written to the
parent directory, lms/static/css. This violates the constraint that
source directories cannot be written to while they are bind-mounted.

To resolve the issue, we merge the contents of lms/static/css/vendor
into common/static/css/vendor. (The directory common/static/css only
contains sources: no CSS is outputted there.)

Now, common/static/css only contains sources for the Sass build, and
lms/static/css is only used for output.

More details: openedx#32835
Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/edx-platform Aug 31, 2023
…ies/[js|css]

To support our old RequireJS-based frontends, a post-install hook on npm
clean-install copies certain JS & CSS modules into
common/static/common/[js|css]/vendor. However, the copied modules'
ancestor directory, common/static/common, is a source for the Webpack
build that we need to bind-mount. This violates the constraint that we
must not bind-mount items that were previously modified by the build.

To resolve the issue, we relocate common/static/common/[js|css]/vendor
to new directories, common/static/node_copies/[js|css]. We provide a
symlink from the original location to a new location.

Now, common/static/common and common/static/node_copies can both be used
as inputs to the Webpack build, but the latter will not be clobbered
when we bind-mount the former. Furthermore, the new directory name
("node_copies") is more illustrative than the old one ("vendor").

(Note: I originally attempted to make this change without the symlink--I
updated all references to the old path to the new path. I struggled for
several hours to get tests passing for JS using RequireJS, and concluded
that I don't understand our RequireJS setup well enough to safely make
that change.)

More details: openedx#32835
Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 31, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal?
* Deprecate patches that no longer exist or have changed.

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Aug 31, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal?
* Deprecate patches that no longer exist or have changed.

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Sep 1, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal?
* Deprecate patches that no longer exist or have changed.

Part of: https://github.com/openedx/wg-developer-experience/issues/166
kdmccormick referenced this issue in kdmccormick/tutor Sep 1, 2023
TODOs:
* See if we can reduce the huge mount blocks a bit.
* Merge edx-platform asset folder changes instead of patching in a PR.
* Add changelog entry.
* Test more thoroughly.
* Circulate a TEP or some other form of proposal?
* Deprecate patches that no longer exist or have changed.

Part of: https://github.com/openedx/wg-developer-experience/issues/166
moonesque referenced this issue in edSPIRIT/tutor Nov 20, 2023
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
openedx-unsupported/wg-developer-experience#71
openedx-unsupported/wg-developer-experience#66
https://github.com/openedx/wg-developer-experience/issues/166
@kdmccormick kdmccormick transferred this issue from openedx-unsupported/wg-developer-experience Mar 28, 2024
@kdmccormick kdmccormick changed the title Speed up Tutor image builds for developers Optimize Tutor's openedx image Mar 28, 2024
@kdmccormick kdmccormick added tutor Requires a change to Tutor and removed epic Large unit of work, consisting of multiple tasks labels Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tutor Requires a change to Tutor
Projects
No open projects
Status: In Progress
Development

No branches or pull requests

1 participant