Use Docker for local development environment

[billglover]

Implement #41 by:

• Providing docker-compose up as a way of instantiating a local development environment
• Providing draft instructions for a new user to get an environment up and running

A couple of known issues:

• Static files are not served by the Django app. These should be moved to nginx.
• Migrations are run on start-up. We should provide a way to trigger these on demand.
• Tests are not run. We should provide a way to trigger these on demand.

I'm creating this pull request for discussion. I encourage developers to clone my fork and experiment with the development experience so that we can get some feedback before this is merged.

[lpatmo]

🎉 🎉
Instructions for how to fork this PR to test out docker:

• git remote add billglover https://github.com/billglover/django-concept.git
• git fetch billglover
• git checkout billglover/master

[lpatmo]

@billglover I tested it out by forking your fork into my personal account, and running docker-compose up looked really promising! :D Then I ran into this:

I believe it'll be fixed if you pull in the latest master changes -- e.g.git pull upstream master and git push origin master (assuming upstream points to the original CB django-concept). Do you mind trying that when you get a chance? There'll be a conflict in README.md (sorry about that); feel free to remove my version.

(I tried that on my personal fork and re-ran docker-compose up, and got this error:
-- which I think is because we have django-taggit==1.2.0 accidentally missing from requirements/local.txt)

[billglover]

I'll give it a go shortly. Dockerfile does include the pip install step. See: Dockerfile#L10.

I can see there is a conflict with the README, but I'll have a go at addressing this.

[lpatmo]

Update: actually (in an attempt to make it easier) I submitted a PR to your fork with the latest as well: https://github.com/billglover/django-concept/pull/1

(Thanks. And yeah, I noticed Dockerfile#L10 about 15 seconds after I submitted my original post 😅)

[billglover]

I've merged the migrations, along with your PR. I've just run this on a clean machine and it seemed to stand-up CBv3 correctly. Would appreciate a couple of other people running through this and confirming it all works as expected.

[MR-R0b0t0]

I am not sure if this is intended or not but I see a couple of things.

1. The folder structure seems odd where we have the same app folder nested inside with the same name:

2. The home page of the project after running docker-compose up is the following:

Is this intended?

[billglover]

I’m not sure either are desirable but neither are the result of using Docker for local development. Static content does need to be addressed in that we need to make a decision on how this is served.

[lpatmo]

@d3vild06 A belated thanks for testing this out!

The folder structure seems odd where we have the same app folder nested inside with the same name:

That's a norm for Django, heh. One of the things that always bothered me. The project is called a certain name, and then we create an app inside it with the same name.

The home page of the project after running docker-compose up is the following:

Ah, good pointing out -- I'll file an issue about adding some content to that Django home page to explain the relevant URLs after loading up the app -- namely /api/v1 and /admin at this point, for example. We should make it clear that the Django project is mainly the API work that the React front-end will feed from.

[billglover]

When running on Docker Desktop for Windows we have reports of the following error message:

ERROR: for db  Cannot create container for service db: C: drive is not shared. Please share it in Docker for Windows Settings
ERROR: Encountered errors while bringing up the project.

[billglover]

The documentation here indicates that you need to specify which drives you want to use with Docker in the settings of the Docker App. This appears to be unique to Windows and is probably a result of the way virtualisation is implemented.

https://docs.docker.com/docker-for-windows/

To use volumes with Docker, users will need to explicitly share drives with the Docker App. For must users this will be the C drive but this is not guaranteed.

[bengineerdavis]

@Bill @lpatmo @d3vild06 Django can accommodate this change, if desired, and there's no penualty, should we want to move everything up a directory. Typically you can remove the extra nesting with (in a venv/container config file) python django-admin startproject . Adding the . is an optional argument for the django admin commands.

[bengineerdavis]

@billglover, per Windows setup, maybe it should be explicit--if confirmed through tests--that setting a separate directory from the C or main directory for Docker is recommended to avoid system complications for Win10.

[nuageklow]

Testing was performed on Windows 10 Pro with docker installed only based on the setup instructions.

edit:
@bengineerdavis I have cloned all the files from the repo which is a django project. I have also tried to run docker-compose up with django-installed venv activated and the same error has occurred as well. I hope I interpreted your suggestion correctly (please let me know if I didn't)

[billglover]

@nuageklow are you able to include a copy of the error you are seeing in your comment so that I can make sure we address it.

[billglover]

I think I’ve found something that might make it easier for Windows users. If possible I need to be more explicit with referencing the current working directory. Will try and find a windows machine to test this on.

[nuageklow]

@billglover

ERROR: for db Cannot create container for service db: C: drive is not shared. Please share it in Docker for Windows Settings ERROR: Encountered errors while bringing up the project.

its the same error as before

[lpatmo]

Django can accommodate this change, if desired, and there's no penualty, should we want to move everything up a directory.
@bengineerdavis Good to know, thanks!

Re: Windows -- if all else fails, Windows users can mainly contribute on the React app (JUST KIDDING! :P) I think we can make a note that Windows users should probably not use the docker-compose approach, and follow a venv + local postgres approach.

[bengineerdavis]

We won't need venv for this, because Docker is in itself isolating python dependencies at runtime. Basically, because Docker containers run in relative isolation from the rest of the operating system--except for when defined explicitly, such as mounted volumes--the pip install -r requirements/local.txt packages will already be isolated from our native install.

[nuageklow]

@nuageklow , can you please run a check on the files related to the docker project and confirm the file permissions and post an image of it here?

@bengineerdavis
is this what you're looking for?

[bengineerdavis]

@nuageklow I think we need to see permissions for the files in the repo you're using to bring up the app with docker-compose, but we also need to find out what the user permissions are for the volume being assigned to the Postgres container. I'm not sure how to specify that just yet (not extremely helpful, I realize).

I would try partitioning another drive on your computer, and then assign to to Docker for Windows, see what the result is. Depending, then we can go about working on file permissions.

[nuageklow]

@bengineerdavis cb is the repo that contains docker-compose.yaml.

let me know if there's a different way i can show you the permission of the repo file.

also please ignore the docker-compose_ori.yaml. its just a copy of the original yaml file as i was trying to different things for to get the db container created.

[nuageklow]

i've figured out the solution!

the root of the issue for windows env is directory of the volume like everyone is indicating. apparently all we need to do is to change the the volume command from ./db_data:/var/lib... to db_data:/var/lib... in the yaml file (i guess this is the way windows indicating folder location..?)

also windows requires a longer (detailed) command to call out a program, for example if i wanna pull psql i'll need to type:
docker exec -it (container name) -d (database) -U (userid) -W e.g. docker exec -it db -d codebuddies -U babyyoda -W

oh also since ./ is removed C drive doesnt need to be shared to install stuff within docker and it still worked properly.

can anyone please tell me how to run python manage.py shell (or any other regular django command) with docker...? i would like to see if i can run the django app without issue.

i really hope that solves everything on the window's side lol thank you everyone for bearing with me and testing out the windows env.

[bengineerdavis]

@billglover confirmed to work on Fedora 30 (Linux) OS with your setup using http://localhost:8000/

Some error messages and adjustments observed:

1. I first ran docker-compose up and got these error messages:

WARNING: Image for service app was built because it did not already exist. To rebuild this image you must use docker-compose build or docker-compose up --build.
Pulling nginx (nginx:1.17-alpine)...
1.17-alpine: Pulling from library/nginx
89d9c30c1d48: Already exists
534c802c70da: Pulling fs layer
ERROR: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/36/36189e6707f43e39466f047ff9564fa55e217821aa7e2326b3f3c32166ca63b0/data?verify=1579585987-mf2TJMMidmlsTuxIEmAY4TkyK%2BI%3D: dial tcp: lookup production.cloudflare.docker.com on 192.168.1.1:53: read udp 192.168.1.7:47407->192.168.1.1:53: i/o timeout

2. After I ran docker-compose build and then up, application ran. I then came across an error message for the manage service container:

django-concept_manage_1 exited with code 0

[bengineerdavis]

Did not mean to close this issue!

[lpatmo]

can anyone please tell me how to run python manage.py shell (or any other regular django command) with docker...? i would like to see if i can run the django app without issue.

@nuageklow Can you try docker-compose run --rm manage shell?

[sunu]

Some error messages and adjustments observed:

1. I first ran docker-compose up and got these error messages:

WARNING: Image for service app was built because it did not already exist. To rebuild this image you must use docker-compose build or docker-compose up --build.
Pulling nginx (nginx:1.17-alpine)...
1.17-alpine: Pulling from library/nginx
89d9c30c1d48: Already exists
534c802c70da: Pulling fs layer
ERROR: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/36/36189e6707f43e39466f047ff9564fa55e217821aa7e2326b3f3c32166ca63b0/data?verify=1579585987-mf2TJMMidmlsTuxIEmAY4TkyK%2BI%3D: dial tcp: lookup production.cloudflare.docker.com on 192.168.1.1:53: read udp 192.168.1.7:47407->192.168.1.1:53: i/o timeout

This looks like a network timeout on docker registry's side. It should fix itself.

1. After I ran docker-compose build and then up, application ran. I then came across an error message for the manage service container:

django-concept_manage_1 exited with code 0

This is kind of expected because the manage container is not exactly a log running service. If this is somewhat confusing, then we can write a Makefile command to start only the services needed. We can also create Make commands for common operations like running migrations or getting into the django shell so that people don't have to remember the long docker-compose commands. eg: make shell instead of docker-compose run --rm manage shell

[sunu]

@billglover Just wanted to point out that we have all these named volumes defined but we're not really using these. Changing ./db_data to just db_data on line 19, ./nginx to nginx on line 55 and so on will make sure we use named volumnes instead of bind mounts. This should fix our problems on Windows too.

[lpatmo]

@sunu Are you saying that if @nuageklow makes only those changes in the docker-compose.yaml file locally, it would probably work on her machine?

[nuageklow]

@lpatmo i actually posted the same solution lastnight earlier on this same thread..i guess i worded it in a different way..?
#44 (comment)

but yes it worked and everything was installed properly.

for docker-compose run --rm manage shell command it didnt work. i'll find a way and post it if i have a solution.

[lpatmo]

Oh sorry, I completely forgot about your comment from yesterday! Too many posts 😅

So the remaining items to do are:
1/ Update the docker file in this PR
2/ Figure out the shell command
3/ sunu's makefile idea in a separate PR (maybe)?

[billglover]

Slight nuance... making the change and removing the ./ to use volume mounts means volumes cannot be mounted from within the project folder and so code doesn't get mapped through to the running application. This means development on the host isn't reflected in the running application so won't work for us.

I'm looking through documentation now to see if we have to drop back to bind mounts to achieve the set-up we are working towards.

[nuageklow]

@sunu @billglover do you think it would be something windows specific only? i was sort of worried abt things might not be placed properly even thought it seemed like it ran smoothly.

also any expected end result be included would be ideal (e.g. how to define the volume is installed properly) so that a newbie like me would know how everything should look like if everything is mounted properly.

update: please disregard this reply i was writing this before bill's new comment #44 (comment). im gonna try things out again tonight / tomorrow. please let me know if u've found a new solution as well @sunu i can try it on my windows env :).

[billglover]

Working on an update today.

Makefile (I'm a big fan) should be a separate PR as we should discuss how Windows users will get the same (or equivalent) experience.

[billglover]

This post goes some way to explaining the behaviour we are seeing here: Define named volume with host mount in the docker compose file.

Unfortunately, I think we need to make bind mounts work with Windows.

[sebbel]

what is this package used for?

[lpatmo]

@sebbel django-taggit is a tag management library that makes creating a global tagging system easier. Docs here: https://django-taggit.readthedocs.io/en/latest/

[sebbel]

so props to @billglover. I could instantly contribute to the repository with the help of this docker setup. kudos!

[billglover]

Confirmed working on Windows 10 Pro. I have been able to complete the following:

• start the application
• access the admin interface in the browser
• make changes to the template and see them reflected in the browser
• log in to Adminer and view database schema
• log in to the Mailhog interface
• run Django manage.py commands

I propose we run the fix for #53 once this is merged.

If anyone would like a walk through of this running on Windows or Mac OS, please message me directly on Slack.

[lpatmo]

Thank you so much for your work on this, @billglover! 🙏 And also to everyone who helped test this out! How do you feel about merging this, and we could file separate issues for:
1/ Windows Home users
2/ creating a makefile

Then:
1/ #53 - clean up .gitignore
2/ Makefile to shorten commands
3/ #54 can be updated

[billglover]

I think this is ready to merge.

I would suggest leaving Windows 10 Home until we have a need for it, but a Makefile and/or script for Windows users would be well worth it. Agree with 1, 2, 3.

[sunu]

Love how well-documented everything is @billglover!

[sunu]

The only remaining nitpick I have is to change restart policy of the containers from always to on-failure. always automatically starts the services even after shutdown and reboot which is not really desirable on local machines; especially laptops.

[billglover]

@sunu This isn’t a nitpick. This is a really good shout. I pulled this together on a desktop that I predominately access remotely. Completely overlooked the fact that the majority will be working on laptops. Will get that changed.