Find a way to cache the npm install step during nox -s dev

[ThomasLaPiana]

Is your feature request related to a specific problem?

nox -s dev takes a really long time....this appears to be primarily due to the fact that npm install step isn't getting cached as node_modules/ is included in the .dockerignore file

Describe the solution you'd like

For the sake of development, we should find a way to cache or use locally installed npm modules to majorly cut down on the time needed to spin up nox -s dev

Describe alternatives you've considered, if any

Not sure what other options might exist yet

Additional context

N/A

[ThomasLaPiana]

@allisonking @ssangervasi do either of you have any thoughts/experience here?

I was thinking we could load local node modules to speed this process up...some kind of check to see if local modules exist than use those, otherwise run an npm install

This might get messy though because we need to guarantee that the versions are the same and such? Maybe npm has some features here I just don't know about that could help solve this problem

[sanders41]

It looks like right now any time any file in clients/admin-ui/ is changed it will cause everything blow to be rebuild and skip cache. Would it solve the issue to do the npm install like with pip? So just copy the package.json file and do the npm install, then later copy the rest of the files?

So

# install node modules
COPY clients/admin-ui/ .
RUN npm install

Changes to

COPY clients/admin-ui/package.json .
RUN npm install

Then later do the COPY clients/admin-ui/ .. This copy would probably have to come in the prod step of the multi stage build to make a difference...assuming it would would at all.

[allisonking]

Hmm yeah I tried to multi-stage it to make the frontend get out of the backend's way, and only build itself on prod, but I guess now that we have fidesctl-ui, it's getting lumped together in the nox command. In the past I've always avoided using docker for developing frontend stuff, and only used it to build the production pages.

Also, for what it's worth, here are the times it took for my last nox -s dev start up:

➜  fides git:(aking-829-db-dataset) nox -s dev
nox > Running session dev
nox > Creating virtual environment (virtualenv) using python3.9 in .nox/dev
nox > docker build --target=dev --tag ethyca/fidesctl:local .
[+] Building 106.3s (22/22) FINISHED                                            
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 37B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 423B                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim-buster  0.7s
 => [internal] load build context                                         14.4s
 => => transferring context: 165.62MB                                     13.8s
 => [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db7  0.0s
 => CACHED [base 2/2] RUN pip install -U pip                               0.0s
 => CACHED [builder  1/14] RUN :     && apt-get update     && apt-get ins  0.0s
 => CACHED [builder  2/14] RUN :     && apt-get update     && apt-get ins  0.0s
 => CACHED [builder  3/14] RUN curl https://packages.microsoft.com/keys/m  0.0s
 => CACHED [builder  4/14] RUN curl https://packages.microsoft.com/config  0.0s
 => CACHED [builder  5/14] RUN :     && apt-get update     && apt-get ins  0.0s
 => CACHED [builder  6/14] COPY dev-requirements.txt dev-requirements.txt  0.0s
 => CACHED [builder  7/14] RUN pip install -r dev-requirements.txt         0.0s
 => CACHED [builder  8/14] COPY requirements.txt requirements.txt          0.0s
 => CACHED [builder  9/14] RUN pip install -r requirements.txt             0.0s
 => CACHED [builder 10/14] COPY optional-requirements.txt optional-requir  0.0s
 => CACHED [builder 11/14] RUN pip install -r optional-requirements.txt    0.0s
 => [builder 12/14] COPY . /fides                                         23.7s
 => [builder 13/14] WORKDIR /fides                                         0.0s
 => [builder 14/14] RUN mkdir -p src/fidesapi/build/static                 0.9s
 => [dev 1/1] RUN pip install -e ".[all,mssql]"                           53.5s
 => exporting to image                                                    12.8s
 => => exporting layers                                                   12.8s
 => => writing image sha256:f9273853b2144bdf8ef68b92b32be9ff94730db6ab9a7  0.0s
 => => naming to docker.io/ethyca/fidesctl:local                           0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker build --target=frontend --tag ethyca/fidesctl:local-ui .
[+] Building 87.3s (10/10) FINISHED                                             
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 37B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 35B                                           0.0s
 => [internal] load metadata for docker.io/library/node:16                 0.8s
 => [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8f  0.0s
 => => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25ecc  0.0s
 => [internal] load build context                                          3.6s
 => => transferring context: 4.81MB                                        3.3s
 => CACHED [frontend 2/5] WORKDIR /fides/clients/admin-ui                  0.0s
 => [frontend 3/5] COPY clients/admin-ui/ .                               15.0s
 => [frontend 4/5] RUN npm install                                         2.2s
 => [frontend 5/5] RUN npm run export                                     57.9s
 => exporting to image                                                     7.7s
 => => exporting layers                                                    7.7s
 => => writing image sha256:7ff4cf97e577247f5ba8c0d7bf9642c518f5fc4e03c87  0.0s 
 => => naming to docker.io/ethyca/fidesctl:local-ui                        0.0s 
                                                                                
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker-compose up -d fidesctl-ui

So the slowest parts for me are:

• pip install -e ".[all,mssql]"
• npm run export

In fact, npm install only took 2s! If we want to speed up npm run export, we just shouldn't run it at all in dev mode since we really don't use it in development (like what I suggested here: #757 (comment))

[sanders41]

@allisonking out of curiosity did npm install run from cache? That would make a big difference if it did.

[ThomasLaPiana]

@allisonking good catch there! definitely would be low-hanging fruit to move npm run export to another stage

I'm also a big curious why the pip install took so long, since all of those packages should be cached 🤔

my run times:

docker build --target=dev --tag ethyca/fidesctl:local .
[+] Building 53.1s (22/22) FINISHED
 => [internal] load build definition from Dockerfile                          0.1s 
 => => transferring dockerfile: 32B                                           0.0s
 => [internal] load .dockerignore                                             0.1s 
 => => transferring context: 35B                                              0.0s 
 => [internal] load metadata for docker.io/library/python:3.8-slim-buster     1.5s 
 => [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a  0.0s
 => [internal] load build context                                            12.0s 
 => => transferring context: 100.50MB                                        11.5s 
 => CACHED [base 2/2] RUN pip install -U pip                                  0.0s
 => CACHED [builder  1/14] RUN :     && apt-get update     && apt-get instal  0.0s 
 => CACHED [builder  2/14] RUN :     && apt-get update     && apt-get instal  0.0s 
 => CACHED [builder  3/14] RUN curl https://packages.microsoft.com/keys/micr  0.0s 
 => CACHED [builder  4/14] RUN curl https://packages.microsoft.com/config/de  0.0s 
 => CACHED [builder  5/14] RUN :     && apt-get update     && apt-get instal  0.0s 
 => CACHED [builder  6/14] COPY dev-requirements.txt dev-requirements.txt     0.0s 
 => CACHED [builder  7/14] RUN pip install -r dev-requirements.txt            0.0s 
 => CACHED [builder  8/14] COPY requirements.txt requirements.txt             0.0s 
 => CACHED [builder  9/14] RUN pip install -r requirements.txt                0.0s 
 => CACHED [builder 10/14] COPY optional-requirements.txt optional-requireme  0.0s 
 => CACHED [builder 11/14] RUN pip install -r optional-requirements.txt       0.0s 
 => [builder 12/14] COPY . /fides                                            21.3s 
 => [builder 13/14] WORKDIR /fides                                            0.1s
 => [builder 14/14] RUN mkdir -p src/fidesapi/build/static                    1.2s
 => [dev 1/1] RUN pip install -e ".[all,mssql]"                              10.2s
 => exporting to image                                                        6.6s
 => => exporting layers                                                       6.6s
 => => writing image sha256:25122a8a620a2977cbe534426777033b8440ed853407645e  0.0s
 => => naming to docker.io/ethyca/fidesctl:local                              0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker build --target=frontend --tag ethyca/fidesctl:local-ui .
[+] Building 63.7s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                          0.0s
 => => transferring dockerfile: 32B                                           0.0s 
 => [internal] load .dockerignore                                             0.0s 
 => => transferring context: 35B                                              0.0s 
 => [internal] load metadata for docker.io/library/node:16                    1.1s 
 => [internal] load build context                                             6.7s
 => => transferring context: 4.96MB                                           6.4s 
 => [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9e  0.0s 
 => CACHED [frontend 2/5] WORKDIR /fides/clients/admin-ui                     0.0s
 => [frontend 3/5] COPY clients/admin-ui/ .                                  12.8s 
 => [frontend 4/5] RUN npm install                                           35.2s
 => [frontend 5/5] RUN npm run export                                         0.8s
 => exporting to image                                                        7.0s
 => => exporting layers                                                       7.0s
 => => writing image sha256:15ac5b4abd31efb09c2354ffa4968341b7a5ef60c8a2f6b9  0.0s
 => => naming to docker.io/ethyca/fidesctl:local-ui 

curious to see that we got some drastically different times for some steps, our caches must've been different

[allisonking]

@allisonking out of curiosity did npm install run from cache? That would make a big difference if it did.

How do I tell?

And yeah that's strange we got such different results @ThomasLaPiana 🤔 presumably I work more on the frontend files and you on the backend, but then I'd expect the reverse since we're modifying those files more 😵

[ThomasLaPiana]

some of it might be down to platform differences (i believe mac's run on a VM of some sort, I'm on windows so its running directly on linux subsystem) but I'm not sure what impact that would have functionally on different steps. It is curious though

[sanders41]

How do I tell?

I don't know of a way to know for sure other than to delete the container all together. docker system prune --all -f. Then do the build. That way you are sure it isn't using cache.

I'd be curious to know what results you both get with no cache at all, then with a rebuild right after when there is cache. You would be sure to be in the same states of cache that way so in theory you would see close to the same times. That would at least tell if the initial difference you saw was related to cache.

[ThomasLaPiana]

nox -s clean -> nox -s "build(test)"

nox > docker build --target=prod --tag ethyca/fidesctl:local .
[+] Building 244.9s (30/30) FINISHED
 => [internal] load build definition from Dockerfile                          0.1s
 => => transferring dockerfile: 2.85kB                                        0.0s 
 => [internal] load .dockerignore                                             0.1s 
 => => transferring context: 459B                                             0.0s 
 => [internal] load metadata for docker.io/library/node:16                    1.9s
 => [internal] load metadata for docker.io/library/python:3.8-slim-buster     1.4s 
 => [internal] load build context                                           126.4s
 => => transferring context: 691.02MB                                       126.2s 
 => [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9  33.1s 
 => => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc08  0.0s 
 => => sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf9118 1.21kB / 1.21kB  0.0s
 => => sha256:fa6e77edbfa850fbcdc38800b429b7d89bf5eff0fae858 2.21kB / 2.21kB  0.0s 
 => => sha256:e6b2a509415bb3adc86fc84ed6b3d9d896e40b34f0dd3a 7.74kB / 7.74kB  0.0s 
 => => sha256:ea267e4631a981caf2841a7e9a1faf29cef9d020c437 50.44MB / 50.44MB  7.9s 
 => => sha256:8a014c92148934973210d840dc7cfed53e0afba38d839a 7.86MB / 7.86MB  2.5s 
 => => sha256:293ff1be7001d642a624409e2d5f90e7708ef7e6f1a7 10.00MB / 10.00MB  2.4s 
 => => sha256:4e42533e7311ad0a85fe19e9bc5fe3138f608853eeaa 51.84MB / 51.84MB  8.9s 
 => => sha256:e56e70ed0dfb39711bca67c13b10e546d3d79ce4a 192.49MB / 192.49MB  16.4s 
 => => sha256:43ccf1ebdd00dc1f1037d0133dfa1fd3fc1bab3eea4957 4.20kB / 4.20kB  8.4s 
 => => extracting sha256:ea267e4631a981caf2841a7e9a1faf29cef9d020c4378fc6484  5.0s 
 => => sha256:6804f7f2f678435f42a46264fb87bd9e0c519c7bac0 34.08MB / 34.08MB  11.7s 
 => => sha256:d1a94681f5387b8a5cbf41dc2fb814c9ed1a6a72bb50bd 2.29MB / 2.29MB  9.5s 
 => => sha256:1715a18f9097ff1184d586e695b776f52314dc65329e87ea87 450B / 450B  9.6s 
 => => extracting sha256:8a014c92148934973210d840dc7cfed53e0afba38d839afaa48  0.8s
 => => extracting sha256:293ff1be7001d642a624409e2d5f90e7708ef7e6f1a75f4eb73  0.6s 
 => => extracting sha256:4e42533e7311ad0a85fe19e9bc5fe3138f608853eeaaea70ee0  3.5s 
 => => extracting sha256:e56e70ed0dfb39711bca67c13b10e546d3d79ce4a6b0fa528c  11.6s 
 => => extracting sha256:43ccf1ebdd00dc1f1037d0133dfa1fd3fc1bab3eea495718828  0.0s 
 => => extracting sha256:6804f7f2f678435f42a46264fb87bd9e0c519c7bac07fb1c7a2  1.7s 
 => => extracting sha256:d1a94681f5387b8a5cbf41dc2fb814c9ed1a6a72bb50bd591fa  0.2s 
 => => extracting sha256:1715a18f9097ff1184d586e695b776f52314dc65329e87ea873  0.0s
 => [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a  0.0s
 => CACHED [base 2/2] RUN pip install -U pip                                  0.0s
 => CACHED [builder  1/14] RUN :     && apt-get update     && apt-get instal  0.0s
 => CACHED [builder  2/14] RUN :     && apt-get update     && apt-get instal  0.0s
 => CACHED [builder  3/14] RUN curl https://packages.microsoft.com/keys/micr  0.0s
 => CACHED [builder  4/14] RUN curl https://packages.microsoft.com/config/de  0.0s
 => CACHED [builder  5/14] RUN :     && apt-get update     && apt-get instal  0.0s
 => CACHED [builder  6/14] COPY dev-requirements.txt dev-requirements.txt     0.0s
 => CACHED [builder  7/14] RUN pip install -r dev-requirements.txt            0.0s 
 => CACHED [builder  8/14] COPY requirements.txt requirements.txt             0.0s
 => CACHED [builder  9/14] RUN pip install -r requirements.txt                0.0s
 => CACHED [builder 10/14] COPY optional-requirements.txt optional-requireme  0.0s
 => CACHED [builder 11/14] RUN pip install -r optional-requirements.txt       0.0s
 => [builder 12/14] COPY . /fides                                            14.4s
 => [builder 13/14] WORKDIR /fides                                            0.1s
 => [builder 14/14] RUN mkdir -p src/fidesapi/build/static                    0.8s
 => [prod 1/3] RUN python setup.py sdist                                      1.7s
 => [prod 2/3] RUN pip install dist/fidesctl-*.tar.gz                        13.1s
 => [frontend 2/5] WORKDIR /fides/clients/admin-ui                            0.3s
 => [frontend 3/5] COPY clients/admin-ui/ .                                   6.3s
 => [frontend 4/5] RUN npm install                                            2.5s
 => [frontend 5/5] RUN npm run export                                        58.6s
 => [prod 3/3] COPY --from=frontend /fides/clients/admin-ui/out/ /fides/src/  0.1s
 => exporting to image                                                       14.7s
 => => exporting layers                                                      14.6s
 => => writing image sha256:b862cea1d9110e72a07c2392a6b6ba986ed4a6c79d4e1f00  0.0s
 => => naming to docker.io/ethyca/fidesctl:local

[sanders41]

My results on Mac with the same commands. Interesting yours still mentions cache.

nox > Running session build(test)
nox > Creating virtual environment (virtualenv) using python in .nox/build-test
nox > docker build --target=prod --tag ethyca/fidesctl:local .
[+] Building 593.9s (30/30) FINISHED                                                             
 => [internal] load build definition from Dockerfile                                        0.0s
 => => transferring dockerfile: 2.74kB                                                      0.0s
 => [internal] load .dockerignore                                                           0.0s
 => => transferring context: 423B                                                           0.0s
 => [internal] load metadata for docker.io/library/node:16                                  1.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim-buster                   1.0s
 => [internal] load build context                                                          23.0s
 => => transferring context: 494.75MB                                                      22.9s
 => [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de60  7.0s
 => => resolve docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de607b8b8  0.0s
 => => sha256:6258dcdb5fea7b710bfcfc3c889e022e4c6e9dd0ea962cfa73fbc130eff2 1.37kB / 1.37kB  0.0s
 => => sha256:8efad32061c2124947b46db2be6969b6698a00f374fbd57048fc2f701ac6 7.53kB / 7.53kB  0.0s
 => => sha256:65db770a0126c5f032de607b8b8ae7207ec2adc17420e54af4d129b1d3b0 1.86kB / 1.86kB  0.0s
 => => sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f47 11.29MB / 11.29MB  4.8s
 => => sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08 233B / 233B  4.9s
 => => extracting sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f4797e53d   0.8s
 => => sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d 3.17MB / 3.17MB  5.3s
 => => extracting sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08   0.0s
 => => extracting sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d467e   0.9s
 => [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37  25.6s
 => => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf  0.0s
 => => sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1f 49.23MB / 49.23MB  2.7s
 => => sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27 7.72MB / 7.72MB  1.1s
 => => sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf911888ff13c7622346 1.21kB / 1.21kB  0.0s
 => => sha256:407013b47e765f5e900729141d6668a3588b484af4082b232e586041e5bb 2.21kB / 2.21kB  0.0s
 => => sha256:90930c7a6dd7f8531d230a2206f35582a1ac1bfb480063ca55dc4e4899dc 7.75kB / 7.75kB  0.0s
 => => sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b6 9.77MB / 9.77MB  0.7s
 => => sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6 52.17MB / 52.17MB  3.5s
 => => sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5 184.07MB / 184.07MB  8.8s
 => => extracting sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1fd5a234   3.7s
 => => sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42 4.09kB / 4.09kB  2.8s
 => => sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05b 34.22MB / 34.22MB  5.2s
 => => sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80 2.29MB / 2.29MB  3.7s
 => => sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482 451B / 451B  3.8s
 => => extracting sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27a5f1   0.5s
 => => extracting sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b65f3a   0.6s
 => => extracting sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6f415d6   4.7s
 => => extracting sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5f3f0afe7  11.2s
 => => extracting sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42a86e   0.0s
 => => extracting sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05ba8417a   0.9s
 => => extracting sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80185b   0.1s
 => => extracting sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482   0.0s
 => [base 2/2] RUN pip install -U pip                                                      26.8s
 => [frontend 2/5] WORKDIR /fides/clients/admin-ui                                          0.6s
 => [frontend 3/5] COPY clients/admin-ui/ .                                                 3.6s
 => [frontend 4/5] RUN npm install                                                         22.8s
 => [builder  1/14] RUN :     && apt-get update     && apt-get install     -y --no-instal  89.8s
 => [frontend 5/5] RUN npm run export                                                      49.2s
 => [builder  2/14] RUN :     && apt-get update     && apt-get install     -y --no-instal  16.5s
 => [builder  3/14] RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key a  6.4s
 => [builder  4/14] RUN curl https://packages.microsoft.com/config/debian/10/prod.list | t  0.6s
 => [builder  5/14] RUN :     && apt-get update     && apt-get install     -y --no-instal  20.8s
 => [builder  6/14] COPY dev-requirements.txt dev-requirements.txt                          0.0s
 => [builder  7/14] RUN pip install -r dev-requirements.txt                                76.3s
 => [builder  8/14] COPY requirements.txt requirements.txt                                  0.0s
 => [builder  9/14] RUN pip install -r requirements.txt                                   212.7s
 => [builder 10/14] COPY optional-requirements.txt optional-requirements.txt                0.0s
 => [builder 11/14] RUN pip install -r optional-requirements.txt                           81.6s
 => [builder 12/14] COPY . /fides                                                           5.5s
 => [builder 13/14] WORKDIR /fides                                                          0.0s
 => [builder 14/14] RUN mkdir -p src/fidesapi/build/static                                  0.3s
 => [prod 1/3] RUN python setup.py sdist                                                    3.1s
 => [prod 2/3] RUN pip install dist/fidesctl-*.tar.gz                                      37.9s
 => [prod 3/3] COPY --from=frontend /fides/clients/admin-ui/out/ /fides/src/fidesapi/build  0.1s
 => exporting to image                                                                      7.2s
 => => exporting layers                                                                     7.2s
 => => writing image sha256:2581a16d6683a60685240853568b0f8f1ce58b2829dc6f2b98a78dfef5b1ca  0.0s
 => => naming to docker.io/ethyca/fidesctl:local                                            0.0s

[ThomasLaPiana]

yeah that seems kinda weird...

oh well, sounds like this PR will just be some tinkering around stream-lining whatever parts of the build that we can, specifically for dev related commands

[allisonking]

Here's mine after docker system prune --all -f then nox -s dev

➜  fides git:(aking-829-db-dataset) nox -s dev
nox > Running session dev
nox > Creating virtual environment (virtualenv) using python3.9 in .nox/dev
nox > docker build --target=dev --tag ethyca/fidesctl:local .
[+] Building 637.0s (22/22) FINISHED                                                                  
 => [internal] load build definition from Dockerfile                                             0.1s
 => => transferring dockerfile: 2.74kB                                                           0.0s
 => [internal] load .dockerignore                                                                0.0s
 => => transferring context: 423B                                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim-buster                        1.1s
 => [internal] load build context                                                               32.1s
 => => transferring context: 769.18MB                                                           31.8s
 => [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de607b8b8  7.0s
 => => resolve docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de607b8b8ae720  0.0s
 => => sha256:b0bff26542d3686b2ab27a0717c161d81eaf10369688d0e880f1c646c3dd7163 2.78MB / 2.78MB   0.9s
 => => sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f4797e53 11.29MB / 11.29MB  3.7s
 => => sha256:65db770a0126c5f032de607b8b8ae7207ec2adc17420e54af4d129b1d3b0745a 1.86kB / 1.86kB   0.0s
 => => sha256:6258dcdb5fea7b710bfcfc3c889e022e4c6e9dd0ea962cfa73fbc130eff2c174 1.37kB / 1.37kB   0.0s
 => => sha256:8efad32061c2124947b46db2be6969b6698a00f374fbd57048fc2f701ac6cbfc 7.53kB / 7.53kB   0.0s
 => => sha256:824b15f81d6568adc139263c39805e52d9880758b907f40144bbb1938ca5932 27.14MB / 27.14MB  2.8s
 => => sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08 233B / 233B       1.0s
 => => sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d467e 3.17MB / 3.17MB   1.7s
 => => extracting sha256:824b15f81d6568adc139263c39805e52d9880758b907f40144bbb1938ca59323        2.2s
 => => extracting sha256:b0bff26542d3686b2ab27a0717c161d81eaf10369688d0e880f1c646c3dd7163        0.2s
 => => extracting sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f4797e53d        0.7s
 => => extracting sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08        0.0s
 => => extracting sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d467e        0.5s
 => [base 2/2] RUN pip install -U pip                                                           34.0s
 => [builder  1/14] RUN :     && apt-get update     && apt-get install     -y --no-install-rec  85.3s
 => [builder  2/14] RUN :     && apt-get update     && apt-get install     -y --no-install-rec  15.8s 
 => [builder  3/14] RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -   6.6s 
 => [builder  4/14] RUN curl https://packages.microsoft.com/config/debian/10/prod.list | tee /e  0.6s 
 => [builder  5/14] RUN :     && apt-get update     && apt-get install     -y --no-install-rec  28.1s 
 => [builder  6/14] COPY dev-requirements.txt dev-requirements.txt                               0.0s 
 => [builder  7/14] RUN pip install -r dev-requirements.txt                                     83.4s 
 => [builder  8/14] COPY requirements.txt requirements.txt                                       0.0s 
 => [builder  9/14] RUN pip install -r requirements.txt                                        221.3s 
 => [builder 10/14] COPY optional-requirements.txt optional-requirements.txt                     0.0s 
 => [builder 11/14] RUN pip install -r optional-requirements.txt                                90.1s
 => [builder 12/14] COPY . /fides                                                                7.2s
 => [builder 13/14] WORKDIR /fides                                                               0.0s
 => [builder 14/14] RUN mkdir -p src/fidesapi/build/static                                       0.3s
 => [dev 1/1] RUN pip install -e ".[all,mssql]"                                                 47.4s
 => exporting to image                                                                           8.4s
 => => exporting layers                                                                          8.4s
 => => writing image sha256:0d214129491d1273ff68d0a4f9843efaa733024cd18f10afa11f8f528a6e05e6     0.0s
 => => naming to docker.io/ethyca/fidesctl:local                                                 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker build --target=frontend --tag ethyca/fidesctl:local-ui .
[+] Building 99.5s (10/10) FINISHED                                                                   
 => [internal] load build definition from Dockerfile                                             0.0s
 => => transferring dockerfile: 37B                                                              0.0s
 => [internal] load .dockerignore                                                                0.0s
 => => transferring context: 35B                                                                 0.0s
 => [internal] load metadata for docker.io/library/node:16                                       1.0s
 => [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aec  39.0s
 => => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf91188  0.0s
 => => sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf911888ff13c76223467466 1.21kB / 1.21kB   0.0s
 => => sha256:90930c7a6dd7f8531d230a2206f35582a1ac1bfb480063ca55dc4e4899dc7674 7.75kB / 7.75kB   0.0s
 => => sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b65f3a 9.77MB / 9.77MB   2.0s
 => => sha256:407013b47e765f5e900729141d6668a3588b484af4082b232e586041e5bbd036 2.21kB / 2.21kB   0.0s
 => => sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1fd5a2 49.23MB / 49.23MB  12.7s
 => => sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27a5f1 7.72MB / 7.72MB   2.9s
 => => sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6f415d 52.17MB / 52.17MB  7.8s
 => => sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5f3f0 184.07MB / 184.07MB  24.1s
 => => sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42a86e 4.09kB / 4.09kB   7.9s
 => => sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05ba841 34.22MB / 34.22MB  11.9s
 => => sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80185b 2.29MB / 2.29MB  12.3s
 => => sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482 451B / 451B      12.4s
 => => extracting sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1fd5a234        4.4s
 => => extracting sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27a5f1        1.2s
 => => extracting sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b65f3a        0.6s
 => => extracting sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6f415d6        4.9s
 => => extracting sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5f3f0afe7       11.8s
 => => extracting sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42a86e        0.1s
 => => extracting sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05ba8417a        1.3s
 => => extracting sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80185b        0.1s
 => => extracting sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482        0.0s
 => [internal] load build context                                                               35.9s
 => => transferring context: 724.99MB                                                           35.6s
 => [frontend 2/5] WORKDIR /fides/clients/admin-ui                                               0.7s
 => [frontend 3/5] COPY clients/admin-ui/ .                                                      6.7s
 => [frontend 4/5] RUN npm install                                                               1.8s
 => [frontend 5/5] RUN npm run export                                                           44.9s
 => exporting to image                                                                           5.4s
 => => exporting layers                                                                          5.4s
 => => writing image sha256:218d35b0efe575e224c435780fe3c34d91828b8f0ce64ac338d3fdafe6c58a34     0.0s
 => => naming to docker.io/ethyca/fidesctl:local-ui                                              0.0s

[sanders41]

I never noticed this before but since we started looking at this I see you can tell if the layer used cache if it starts with CACHED. So in both of your runs the npm install was done without cache, but it is always way faster than both Thomas' and mine. No idea why that would be.

[ThomasLaPiana]

1.8s doesn't seem like enough time to actually install all of those modules? there's some kind of black magic cache going on or something 😂 i don't know how else that can be possible

[ThomasLaPiana]

@allisonking I'm going to defer on this one until you finish up the frontend packaging PR since I don't want to potentially create conflicts

[ThomasLaPiana]

@NevilleS fixed this 🚀