Mutagen optimization #63

yeegor · 2020-02-19T14:04:44Z

Bring the project up using Mutagen.
TODO: in closest time make docker-compose files to be able to bring the project up without Mutagen.

building PWA theme.

- Add volume core-frontend-data for core frontend container - Implement mutagen configuration for both core and frontend dc configs - FIx node container not having wget - Keep maildev configuration in local setup only

Make them be mounted into frontend container on deploy instead of copied on build

Optimize sync making it two-way-resolved Propagate www-data as default file owner

to core compose

to corresponding compose file

yeegor · 2020-02-21T14:21:02Z

For now I am unable to install backend module using this setup. Investigating.

- remove nginx session by introducing named volume - extend mutagen permission config with default file mode

build/php/Dockerfile

deploy/shared/conf/local-ssl/ca.conf

deploy/start-demo.sh

deploy/start.sh

alfredsgenkins · 2020-02-24T04:40:11Z

Please try adding container_name: my-web-container for all setvices docker-compose.mutagen.yml or in docker-compose.local.yml to resolve naming issue. Note: this prevents from scaling.

alfredsgenkins · 2020-02-24T04:41:49Z

Please link a doc for this, otherwise Ilja won't review this.

alfredsgenkins · 2020-02-24T04:58:57Z

Try specifying user in docker-compose like this: https://docs.docker.com/compose/compose-file/#userns_mode#domainname-hostname-ipc-mac_address-privileged-read_only-shm_size-stdin_open-tty-user-working_dir

build/frontend/start.sh

deploy/start-demo.sh

deploy/start-with-frontend.sh

deploy/start.sh

eli-l · 2020-02-24T12:02:48Z

docker-compose.yml

@@ -53,7 +54,7 @@ services:
    volumes:
      - ./deploy/shared/conf/nginx/magento.conf:/etc/nginx/conf.d/default.conf
      - ./deploy/shared/conf/nginx/cache-router.conf:/etc/nginx/conf.d/cache-router.conf
-      - app-data:/var/www/public
+      - nginx-data:/var/www/public/pub


What's the reason of introducing new volume? Is that mutagen limitation?

It is not mutagen-related, but apparently it is faster to sync many small named volumes with their containers than one huge app-data volume with many containers.

It is not the only volume introduced, see core-frontend-data.
I also wanted to create a frontend-data volume to mount it into the frontend container, but the issues occur because we then would need to mount vendor and app/design/frontend/scandiweb/pwa folders separately and vendor doesn't want to be mounted separately for some reason :(

It is not mutagen-related, but apparently it is faster to sync many small named volumes with their containers than one huge app-data volume with many containers.

Ok, means that you prefer "faster sync" to developer, who is adding additional file and want it to be served directly from nginx, to spend TONS of time to find out it is not mounted to nginx? This approach is wrong.

Current solution considers mounting app-data into nginx container. This way we will not have one more docker-compose file. Is there any difference in mounting app-data volume into nginx or creating a bind mount? Additional sync sessions will be slower than direct named volume mount, should I create them between nginx and local fs instead of mounting app-data volume into nginx?

to resolve container naming issue in yml files

yeegor · 2020-02-24T12:44:37Z

Known issue: Currently I am experiencing problems with configuring Xdebug on my Mac using this configuration.
Investigating.

yeegor · 2020-02-24T13:33:38Z

Try specifying user in docker-compose like this: https://docs.docker.com/compose/compose-file/#userns_mode#domainname-hostname-ipc-mac_address-privileged-read_only-shm_size-stdin_open-tty-user-working_dir

When I refer to user 'user' by name, the container does not start.
When I refer to user 'user' by UID 1000, I get the following error:

mutagen-powered-app | Starting with UID : 0
mutagen-powered-app | useradd: Permission denied.
mutagen-powered-app | useradd: cannot lock /etc/passwd; try again later.
mutagen-powered-app | error: failed switching to "user": unable to find user user: no matching entries in passwd file

eli-l · 2020-02-24T15:25:14Z

build/frontend/Dockerfile

@@ -8,9 +8,4 @@ ARG BASEPATH=/var/www/public
 # Set working directory so any relative configuration or scripts wont fail
 WORKDIR $BASEPATH



Why these scripts were removed? These files contains deployment logic, not application version/dependencies.

Returned a single script to build frontend container

eli-l · 2020-02-24T15:26:37Z

build/frontend/start-core.sh

@@ -1,4 +1,12 @@
 #!/bin/bash
+
+echo "Waiting for initial files to be synced"


I'd be more up to enforce re-building when dependencies are changed, rather than to use such a hacky solution. It is needed not because Docker is not-to-good, but because we are moving away from it's best practices to far.

Do you propose to do install dependencies during the frontend container build and remove npm ci from this script at all, leaving it with a single purpose of launching pm2-runtime? Then we can get rid of these deployment scripts of frontend container and do that with docker-compose file command: ['npm', 'run', 'pm2-watch'] directive?

npm ci was moved to frontend container build script.

build/frontend/start.sh

build/rendertron/Dockerfile

deploy/start-demo.sh

eli-l · 2020-02-24T15:47:14Z

deploy/start-demo.sh

@@ -286,6 +284,12 @@ function exit_catch {

 ### Deploy pipe start

+echo "${blue}${bold}Waiting for Mutagen to sync initial filese${normal}"


Composer.json/lock is always in a container, as these are added during the build.
Does mutagen add additional layer that is empty on top of containers FS, similar to local mount? Even so - this behavior is related only to local.

For mutagen - use another start scripts, hopefully we'll unify it one day.

eli-l

Global comment:

pay attention and rework workdirs, don't rework/redefine them unless absolutely necessary for Mutagen (never?)
source code must be shared between nginx and app containers, no matter how you run it (no mounting, docker native, mutagen), it is done for a reason.
revisit all the changes (all existing files that have been edited by you), and unless this is absolutely good for ALL environments and use cases - move these changes to a separate file (even it is 300 lines long).
remove changes to the existing services, that are not related to your task

eli-l · 2020-02-24T16:20:42Z

deploy/start-demo.sh

@@ -286,6 +284,12 @@ function exit_catch {

 ### Deploy pipe start

+echo "${blue}${bold}Waiting for Mutagen to sync initial filese${normal}"


For mutagen - use another start scripts, hopefully we'll unify it one day.

deploy/start-with-frontend.sh

deploy/start.sh

eli-l · 2020-02-24T16:24:32Z

deploy/start.sh

@@ -264,6 +263,13 @@ function exit_catch {

 ### Deploy pipe start

+echo "${blue}${bold}Waiting for Mutagen to sync initial filese${normal}"


This is the default file - nothing related to local, mutagen or other side-cases must be added here.
create deploy/mutagen directory, put there everything related to Mutagen sync, anyway you have mutagen dedicated docker-compose files, controlling start script is extremely easy there.

eli-l · 2020-02-24T16:25:19Z

docker-compose.core.yml

@@ -14,9 +14,15 @@ services:
        - NODEJS_VERSION=${NODEJS_VERSION}
    volumes:
      - ./src/localmodules/base-theme:/var/www/public
-    command: ["npm", "run", "dev-server-core"]
+      - ./build/frontend/start-core.sh:/start-core.sh


Build file into container, do not mount it from the host.

Move dependency installation to build script Replace deploy script with single command Remove core-frontend-data volume, mount app-data directly Rename start-with-frontend script

Remove waiting for composer.json files in main script

…se into mutagen-optimization

yeegor added 7 commits February 19, 2020 10:03

Fixing certificate revoked on Mac

13db825

Use Mutagen to bring up core config

e72bfd4

Wait for necessary files to be present when

e0da98d

building PWA theme.

Remove whitespaces

0248771

Mac optimizations

b019d5c

- Add volume core-frontend-data for core frontend container - Implement mutagen configuration for both core and frontend dc configs - FIx node container not having wget - Keep maildev configuration in local setup only

Optimize frontend deploy script acquiring process

54ecd38

Make them be mounted into frontend container on deploy instead of copied on build

Add mutagen config without frontend container

e7a2014

Optimize sync making it two-way-resolved Propagate www-data as default file owner

yeegor requested a review from eli-l February 19, 2020 15:51

yeegor added 6 commits February 19, 2020 18:32

Move core-frontend-data volume from main compose

78757ca

to core compose

Fix permission issue in app and nginx containers

bd454be

Rename mutagen compose files to keep originals

c4acedb

Add modified original compose files back

21b99ad

Move core frontend volume declaration

2bf3b94

to corresponding compose file

Fix user being set as file owner on alpha

30fcba8

Refactor sync sessions

e6418e2

- remove nginx session by introducing named volume - extend mutagen permission config with default file mode

alfredsgenkins requested changes Feb 24, 2020

View reviewed changes

build/php/Dockerfile Outdated Show resolved Hide resolved

deploy/shared/conf/local-ssl/ca.conf Outdated Show resolved Hide resolved

deploy/start-demo.sh Outdated Show resolved Hide resolved

deploy/start.sh Outdated Show resolved Hide resolved

eli-l self-assigned this Feb 24, 2020

eli-l added the enhancement New feature or request label Feb 24, 2020

yeegor added 3 commits February 24, 2020 13:50

Revert not working Mac certificate fix

a5e2012

Typo and ping removal

d793ec8

Remove ping from installed packages

d167dfd

eli-l suggested changes Feb 24, 2020

View reviewed changes

Add mutagen-powered-app container name

a3eafa0

to resolve container naming issue in yml files

eli-l suggested changes Feb 24, 2020

View reviewed changes

yeegor added 16 commits February 27, 2020 21:32

Refactor frontend container compose files

932f896

Move dependency installation to build script Replace deploy script with single command Remove core-frontend-data volume, mount app-data directly Rename start-with-frontend script

Mount all app-data into nginx

a713d98

Return magento se:up success check to deploy

90945d5

Update process file names to the relevant ones

ffd8723

Return dependency installation to FE deploy

d4be409

Revert demo deploy script changes

1b7c6b5

Use separate deploy script for Mutagen

532edd7

Remove waiting for composer.json files in main script

Use start-with-mutagen script to wait for fs

2f5058e

Map Mutagen start scripts into containers

8abc4a8

Build scripts into containers instead of map

c1d28a5

Mount app-data into frontend container

765ccc4

Remove mounting start script from host

6944839

Move waiting logic to separate script for FE cont

5188423

Apply proper permissions to waiting script

be2e3d8

Implement script to wait for fs

b9ac967

Merge branch 'mutagen-optimization' of github.com:yeegor/scandipwa-ba…

7cbd8f0

…se into mutagen-optimization

alfredsgenkins changed the base branch from 2.x-stable to 2.x-mutagen April 21, 2020 15:22

alfredsgenkins merged commit ae0df1c into scandipwa:2.x-mutagen Apr 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mutagen optimization #63

Mutagen optimization #63

yeegor commented Feb 19, 2020

yeegor commented Feb 21, 2020

alfredsgenkins commented Feb 24, 2020

alfredsgenkins commented Feb 24, 2020

alfredsgenkins commented Feb 24, 2020

eli-l Feb 24, 2020

yeegor Feb 24, 2020

yeegor Feb 24, 2020

eli-l Feb 24, 2020

yeegor Feb 27, 2020

yeegor commented Feb 24, 2020

yeegor commented Feb 24, 2020 •

edited

Loading

eli-l Feb 24, 2020

yeegor Feb 28, 2020

eli-l Feb 24, 2020

yeegor Feb 27, 2020

yeegor Feb 28, 2020

eli-l Feb 24, 2020

eli-l Feb 24, 2020

eli-l left a comment

eli-l Feb 24, 2020

eli-l Feb 24, 2020

eli-l Feb 24, 2020

		@@ -8,9 +8,4 @@ ARG BASEPATH=/var/www/public
		# Set working directory so any relative configuration or scripts wont fail
		WORKDIR $BASEPATH

		@@ -1,4 +1,12 @@
		#!/bin/bash

		echo "Waiting for initial files to be synced"

		@@ -286,6 +284,12 @@ function exit_catch {

		### Deploy pipe start

		echo "${blue}${bold}Waiting for Mutagen to sync initial filese${normal}"

		@@ -264,6 +263,13 @@ function exit_catch {

		### Deploy pipe start

		echo "${blue}${bold}Waiting for Mutagen to sync initial filese${normal}"

Mutagen optimization #63

Mutagen optimization #63

Conversation

yeegor commented Feb 19, 2020

yeegor commented Feb 21, 2020

alfredsgenkins commented Feb 24, 2020

alfredsgenkins commented Feb 24, 2020

alfredsgenkins commented Feb 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeegor commented Feb 24, 2020

yeegor commented Feb 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eli-l left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeegor commented Feb 24, 2020 •

edited

Loading