Add an option to develop in a reproducible and containerized environment

[KeyWeeUsr]

Adds an option to develop this application in a containerized environment that provides reproducibility between multiple machines with the same change (branch / pull request / ...). In the less common approach it provides a way to develop applications (mainly web) without installing Node.js and maintain multiple dependencies, node_modules, etc on the host machine without an easy way to just burn everything down and start from scratch in case of a wrong approach.

Docker allows kind of git stash and git stash pop behavior with images therefore everything can be properly isolated, the changes can be done via mounting appropriate folders from host machine and in the worst case you can create an image via docker commit in case something went horribly wrong and you want your colleague to check it out on his/her machine (see docker save). Currently the image takes about 855MB on my machine, but that might be later improved.

This way you can have an image for develop, master, fix-12345, etc and also work just fine on Riot X for example if it uses the same repo (not sure, haven't checked out yet). Storage nowadays isn't much of an issue I'd say, however having collisions between deps while working on multiple projects or the need to provide versions of multiple random binaries between colleagues just to reproduce why something does not work is rather inefficient.

Hopefully this will help somebody else other than me :)

---

This PR currently has no changelog labels, so will not be included in changelogs.

A reviewer can add one of: T-Deprecation, T-Enhancement, T-Defect, T-Task to indicate what type of change this is, or add Type: [enhancement/defect/task] to the description and I'll add them for you.

[turt2live]

This is looking in the right direction - thank you! Most of the concerns here are slightly awkward workflow constraints on our end, but should hopefully be achievable.

If you run into problems, visit us in #riot-web:matrix.org (which I think you've found?).

[turt2live]

. o ( We should probably bump this to 11 or 12 soon. Something for a different PR )

[Palid]

Definitely need to bump it to node-14 now, as we don't support anything older than that.

[melroy89]

you can use node:lts-alpine if you want

[turt2live]

meh - this is fine tbh. It shouldn't cause problems, it's just a waste of cycles.

tbh I'm more inclined to leave it there than rip it out for future maintenance things.

[KeyWeeUsr]

I've tried removing it again today and it throws this error:

Step 12/15 : RUN yarn --network-timeout=100000 install
 ---> Running in a026b8e776e8
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
warning Pattern ["gemini-scrollbar@matrix-org/gemini-scrollbar#b302279"] is trying to unpack in the same destination "/usr/local/share/.cache/yarn/v4/npm-gemini-scrollbar-1.4.3/node_modules/gemini-scrollbar" as pattern ["gemini-scrollbar@github:matrix-org/gemini-scrollbar#b302279"]. This could result in non-deterministic behavior, skipping.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "matrix-react-sdk > [email protected]" has unmet peer dependency "emojibase@*".
warning "matrix-react-sdk > [email protected]" has unmet peer dependency "immutable@>=3.8.1".
warning "matrix-react-sdk > [email protected]" has unmet peer dependency "immutable@>=3.8.1".
warning "matrix-react-sdk > [email protected]" has unmet peer dependency "immutable@>=3.0.0".
warning "matrix-react-sdk > [email protected]" has unmet peer dependency "immutable@>=3.8.1".
warning "matrix-react-sdk > diff-dom > [email protected]" has unmet peer dependency "rollup@>=0.66.0 <2".
warning "matrix-react-sdk > slate-react > [email protected]" has unmet peer dependency "immutable@>=3.6.2".
warning "matrix-react-sdk > slate-react > [email protected]" has unmet peer dependency "immutable@>=3.8.1".
warning "matrix-react-sdk > slate-react > [email protected]" has unmet peer dependency "immutable@>=3.8.1".
warning " > [email protected]" has unmet peer dependency "app-builder-lib@~21.2.0".
[4/4] Building fresh packages...
$ yarn clean && yarn build:compile
yarn run v1.17.3
$ rimraf lib webapp electron_app/dist
Done in 0.24s.
yarn run v1.17.3
$ yarn reskindex && babel --source-maps -d lib src
$ reskindex -h src/header
src doesn't exist
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
The command '/bin/sh -c yarn --network-timeout=100000 install' returned a non-zero code: 2

[turt2live]

we should fix that error then - that file definitely exists. Is it just not making its way over to the container?

[KeyWeeUsr]

Eh, this is embarrassing... for some reason it never occurred to me it's a file ^_^" Fixed!

[turt2live]

we should actually be using locally-linked react-sdk and such when doing development, as the bulk of Riot is in the react-sdk.

[KeyWeeUsr]

Working on it 👍

[turt2live]

the variable is still here :(

[KeyWeeUsr]

Yes, it is, otherwise the container would require people to mount files to certain location. Currently it by default pulls the repos, however allows to override this behavior. This way not only you can choose what repo/folder to use, but also allows you to have only this project as editable without even pulling the SDKs. See this change.

[turt2live]

for a development environment where 95% of the app is in the react-sdk, I would definitely expect that people have a local copy of the react-sdk.

[KeyWeeUsr]

The current approach works for both cases and at the same time pulls the app dependencies first which means that there already are dependencies for react-sdk and riot-web already cached in the image itself. That means a user that mounts a locally cloned react-sdk does not have to wait long for the installation because the only thing that's going to install with docker run is riot-web + different versions of deps that are not cached. This is good for switching between branches since docker container by default does not save the changes in its filesystem(image) unless docker commit is used.

[turt2live]

The installation should be pretty quick if the dependencies have already been pulled the first time. Unfortunately the environment doesn't provide much value if most of the app isn't editable.

[KeyWeeUsr]

Docker does not work that way, it does not preserve anything that happens in the container which means every docker run command requires the installation from scratch i.e. pulling each dependency. No quick install because the pull would happen every time. That's why I left the original repositories there so the docker image already contains the cached dependencies. Only image preserves the state, not container. If I remove the repositories from Dockerfile, everything will be slowed down because the whole fetching/hashing/installing/linking/preparing process would need to be executed in full on each docker run. Currently if you do docker build the image (which contains the state used by the container) will have all the deps + a default installation according to that ARGS instruction, however you don't need to care about that because as said in the linked instructions from README.md with docker run you can specify to use custom, locally available, repositories/folders which will then replace the default installation from the image and will be both used for the next installation run with docker run command (installs only the new SDKs, uses cached deps, i.e. the case you describe) and will be editable from the host machine i.e. if I change something in react-sdk and press F5 the changes will be immediately reflected in the container. If anything goes wrong a user can then use docker commit to create a new image from the container and e.g. send it to you/other devs or just share with colleagues.

[turt2live]

/riot/src would still be better, I think.

[KeyWeeUsr]

Working on it! :) Hopefully until the end of the week I'll have more time for testing Docker builds, since this will need to change the folder within Dockerfile (already present in master).

[KeyWeeUsr]

Resolved

[KeyWeeUsr]

@turt2live I've implemented two ways for using a custom SDKs, one in docker build for one-time usage (e.g. cloning from release branch) and one in docker run for editing on-the-go while having riot-web + SDKs on the host machine.

[turt2live]

we're much closer now, thank you! Just a couple things left to resolve.

[jonathancross]

@KeyWeeUsr Has this been abandoned?

[rohith5112]

Ok so u have to describe the issue more for me thank you

[Palid]

@KeyWeeUsr is this still relevant? If yes, ping me and I'll take care of it. :)

[KeyWeeUsr]

@KeyWeeUsr is this still relevant? If yes, ping me and I'll take care of it. :)

Well, afaik it still is, I haven't really seen any alternative provided.

What exactly still needs to be changed in this PR though (except the obvious scripts/docker-link-repos.sh conflict)?

[Palid]

@KeyWeeUsr is this still relevant? If yes, ping me and I'll take care of it. :)

Well, afaik it still is, I haven't really seen any alternative provided.

What exactly still needs to be changed in this PR though (except the obvious scripts/docker-link-repos.sh conflict)?

conflicts and review comments, please resolve them if you think they have been taken care of.

[KeyWeeUsr]

From what I see (1, 2, 3) the only thing is the Node 14.x+ version, no?

[Palid]

From what I see (1, 2, 3) the only thing is the Node 14.x+ version, no?

I think so, to me your explanation sounds good, especially considering that the VSCode's development in docker is so much better now and it'll work much faster than sharing directory on root machine (especially on MacOS :), but just to make sure I'll ping @turt2live too.

[turt2live]

So we've taken another look at this and don't currently have a pressing need for it within the team, but if folks are interested in the sort of benefits this offers then please open a new issue/PR and we can discuss it there.

Thanks for taking a look at this, and apologies for the slow review cycle.