First pass to make webwork run on docker

[xcompass]

Now, people can just run docker-compose up -d to run a local dev environment using docker. No more complicated setup.

This is largely based on http://webwork.maa.org/wiki/Installation_Manual_for_2.12_on_Ubuntu_16.04

[sharumpe]

Glad I searched -- I was just going to see what it would take to do this, so I'm glad to see someone has already solved it. :)

[heiderich]

Looks promising. On Ubuntu 16.04 I had the problem that /dev/tcp was not present. Instead of
while ! timeout 1 bash -c "(cat < /dev/null > /dev/tcp/$WEBWORK_DB_HOST/$WEBWORK_DB_PORT) >/dev/null 2>&1"; do sleep 0.5; done
I used the following in docker-entrypoint.sh:
until nc -z $WEBWORK_DB_HOST $WEBWORK_DB_PORT; do sleep 0.5; done
See https://github.com/heiderich/webwork2/commits/docker_with_netcat for my changes.

Of course this depends on netcat. Is there a more universal approach?

[xcompass]

@heiderich Is it possible your $WEBWOKR_DB_HOST and $WEBWORK_DB_PORT were not set when you try it? I tested command on Ubuntu 16.04, it works fine. I replaced the variables with a known value.

ubuntu@ubuntu-xenial:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
ubuntu@ubuntu-xenial:~$ cat < /dev/null > /dev/tcp/216.58.193.68/80
ubuntu@ubuntu-xenial:~$

[xcompass]

@jwj61 @pstaabp Any plan to merge this? It will be a lot easier for people to setup webwork.

[mgage]

Hi Pan,

@xcompass

Yes. We absolutely intend to merge this PR with docker files.

And thank you very much for creating it. I'm still getting used to the kinds of things that Docker (and Vagrant) can do to make setup easier.

I was planning to include it in 2.14 to be released at the end of the summer because it will take some time on my part to get my head around Docker, test it properly and write up instructions for installing WeBWorK using Docker. I originally wanted to split of 2.13 as a beta branch for testing today -- but it looks like that will have to wait until July 6 since I'll be gone over the long weekend. Once 2.13 is split off we can start pulling new things into develop. the utf8/localization pulls and the webwork3 pulls are a priority but the docker pull is included.

My hope is to split 2.13 off as a beta branch on July 6 or there about and then merge it into master a week or so later.

If you think it's safe and worth it to include Docker immediately in 2.13 we can pull your docker PR before we split off 2.13 but in that case I'll need help in testing and writing the docs for installing WW in Docker.

-- Mike

[xcompass]

@mgage Thanks for the reply. Since this is the first pass, I wouldn't run it in prod. We are using this for development right now. So I expect there will be some changes if you would like to run containerized webwork in prod. (I'm planning to run it in prod in the near future as well). I can help with docs as well.

I think for now, maybe it is better for merge it into develop branch and anyone who is interests can try it out. The problems (if there is any) will pop up and we can fix it before going into master. Would you like me to change the PR to develop branch?

[mgage]

@xcompass That sounds like a good plan and on my end it will be less rushed. Please change the pull request to point to development. It's best if it can be rebased off a recent version of development so that it is easier to check the recent commits. We're trying to keep the commit lines as clean as possible in the network graphs.

[xcompass]

@mgage I cherrypicked the two commits to develop branch. And updated the PR base branch to develop.

[xcompass]

Please feel free to squash merge two commits.

[mgage]

Looks good so far. I'll look more closely and test merging it into develop as soon as I get 2.13 split off the develop branch.

[heiderich]

@xcompass: I do not remember whether $WEBWOKR_DB_HOST and $WEBWORK_DB_PORT were set or not. But I am almost sure that even /dev/tcp did not exist. But docker-entrypoint.sh is executed within the container, right? So it should not depend on the host system. I cannot reproduce it anymore.

[xcompass]

@heiderich /dev/tcp does't exist at all. It is the bash internal thing. You won't find it on the file system. https://www.gnu.org/software/bash/manual/bashref.html#Redirections

[heiderich]

@xcompass Thank you. I was not aware of this bash feature.

[dsteinmo]

I've tested this out and it works as advertised. Significantly improves the development experience, I would recommend it gets merged into develop branch.

[mgage]

I agree that it would be great to have this in development. With the right instructions it will make it easier for more people to check pull requests. Did you check the email feature by any chance? I’m worried that it has not caught the switch from CPAN Mail::Sender to Email::Sender which was made between ver 2.12 and 2.13. I’ll get to this on Thursday if no-one beats me to it. — Mike

…

On Nov 11, 2017, at 2:46 PM, Derek Steinmoeller ***@***.***> wrote: @dsteinmo approved this pull request. I've tested this out and it works as advertised. Significantly improves the development experience, I would recommend it gets merged into develop branch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openwebwork_webwork2_pull_747-23pullrequestreview-2D75940422&d=DwMCaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=C6Pt5AGtImanmAdcooarL-JZO8M5dSFPfs3VweYXYkE&m=X5fO-CoK3q9Jm-T2-Owxtrqb9FyMt89f3iAmX0oyMkg&s=Yr9HtziAvZlNUzTTEgYcIg1c19kYv-PXmRTogVSs1no&e=>, or mute the thread <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABJdfJeS28UKDx6kE6lpzOMEJU0rP5Zuks5s1fmYgaJpZM4KQgG1&d=DwMCaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=C6Pt5AGtImanmAdcooarL-JZO8M5dSFPfs3VweYXYkE&m=X5fO-CoK3q9Jm-T2-Owxtrqb9FyMt89f3iAmX0oyMkg&s=N731Li-jFKOwg1m9wVgzSbjdRERn6mpoQ2OfavUfdog&e=>.

[dsteinmo]

It looks like he pulls in both Mail::Sender Email::Sender::Simple in the Dockerfile in this PR, so it should be good.

I should have clarified that I tested the latest 'docker-with-r' branch here: https://github.com/ubc/webwork2/tree/docker-with-r , since I wanted to see if the R setup could be dockerized as well, which it can. The result of 'docker-compose up' is three simultaneous containers, one for R, one for mariadb, and one for webwork.

We can probably ask if @xcompass can re-open this PR with the latest and greatest that has both Rserve support and we can be sure is compliant with the v2.13 setup.

[xcompass]

I would be really happy to include R. Since R is not standard component in webwork (yet), I didn't include it in this PR. If people are interested in the one with R, I can update this PR. The only differences are the additional Statistics::R::IO perl module and R(Rserve) container.

[mgage]

Hi @xcompass

I've finally had a chance to try this -- starting as a complete newbie about docker.
I've been able to create two containers -- one for the database and the other for webwork.

I added some things to the containers (populated the OPL and pulled the sagefix PR into PG.
(Incidentally, while webwork2/.git is present in the container pg/.git is not. I copied it in using
the dock cp function.

This is all really neat. I notice that you have another version with R now, would it be better to include
that instead of this pull request.

I feel like I've learned enough to understand answers at this point. I've understood how to share
containers individually -- I've failed at trying to share a service that connects the database and webwork containers.

I wanted to share my updated containers with my laptop (combined into a service with a network that connects webwork and the database)

I was able to load the containers into the docker hub: https://hub.docker.com/ . And I manually
copied the docker-compose.yml to a directory on my laptop. Running docker-compose up in that directory brings in both containers and the database stays up, but the webwork container dies. I added Dockerfile and entry-entrypoint.sh to the directory. That changed some things and the database builds nicely but the webwork_app won't build and fails with the error message that it can't find /opt/webwork/webwork2/conf/site.conf.dist (although that file exists in the container).
Perhaps the Dockerfile has to be tweaked somehow in order for this to work?

The containers I built can be found by going to https://hub.docker.com/explore/ and typing michaelegage into the search box.

Thanks for starting this project. And I'm sorry to have taken so long to get into reviewing it myself.
I'm ready to merge this request but I'd also like to give explicit constructions on

1. how to create a local instance of webwork (on mac since that is what i'm using)
2. how to update the OPL
3. how to transfer .git if necessary to .pg (although I think that is a bug)
4. -- next someone would create a couple of courses in their local container
5. this is the part I don't understand yet: how to pass a copy of your local container to another computer or user so that they can test it.

If we can get a workflow like this operating smoothly I think a lot more people will be encouraged to become reviewers.

Thanks.
-- Mike

[mgage]

I've figured out a reasonable workflow -- see "Docker newbie instructions" on the github wiki

https://github.com/openwebwork/webwork2/wiki/Docker-newbie-instructions

It doesn't use containers to hold data yet, but relies on the mount of webwork2 and courses
at webwork2/.data/webwork2 and webwork2/.data/courses. That's good enough for now.

I've had these files on production machines (where docker is not being used) and as expected they cause no trouble.

I hope that the explanations in "Docker newbie instructions" is enough to get people interested.
Further modifications, perhaps taking advantage of the newest Docker features, may make it even easier to set up your own private instance of WeBWorK for testing.

[xcompass]

Hi Mike, thanks for merging the PR. I'll send another PR including R. It will include the external PG and OPL mount as it is for webwork2 so people can update PG and OPL.

I also made a few updates in the newbie instruction. Let me know if you have any question.