POCS in the Dock(er) #778
POCS in the Dock(er) #778
Conversation
* Use apt packages from defined list * Don't use conda (our whole container is our app so using system python is fine) * Astrometry comes from apt; index files will be a docker volume
Base file for running unit could be used with just `docker-compose up` Whereas development can use `docker-compose -f docker-compose.yml -f docker-compose.dev.yaml` which will merely bind the host `$POCS` dir to the container.
This makes it so we can connect across a different network, not just localhost. The main usage will probably be within a private network while running docker so we turn a blind eye to any security.
This presents a number of changes for building and running POCS and friends in a docker environment. As of this PR, starting the environment (via docker-compose) will launch four separate services (container images): * messaging-hub * weather * paws * pocs As of now, it will also map the following directories to the host: * $PANDIR/json_store * $PANDIR/logs * $PANDIR/images * $PANDIR/POCS It also appropriately sets up the internal network that allows the services to communicate with each other over zeromq and exposes port 8080 on the host machine for reaching PAWS. The environment can be started with a helper script: ```bash $POCS/scripts/run_pocs_docker.sh [help|jupyterlab] ``` As of now there is one additonal option that can be passed (`jupyterlab`) which will start a jupyterlab environment within the running pocs service, exposing port 8888 on the host for access. The POCS repo is installed in development mode (i.e. `pip install -e .`) so that changes to the `$POCS` dir on the host will be reflected in the running service. To do still: * Environment sensors a la the weather container. * Convenience script to make it easier for non-technical users to install docker, authenticate, pull images, etc. * Lots more I'm sure. Note: this runs all of the docker images in `privileged` mode, meaning they have access to the host. Also: * Dockerfile to create a `pocs-base` image that contains the POCS repository installed on an ubuntu 18.04 base image. This includes an install of all the dependencies, including astrometry.net. Some of the apt-packages have been cleaned up or changed as part of this PR. * docker-compose.yaml files for easily starting all of the containers and helpers. * Bash script for starting environments. * Helper scripts for starting services. The POCS script (`$POCS/scripts/docker/run_pocs.sh`) still needs some work. * Small README updates (more likely to come).
Codecov Report
@@ Coverage Diff @@
## develop #778 +/- ##
===========================================
- Coverage 81.49% 79.88% -1.61%
===========================================
Files 69 68 -1
Lines 5581 5519 -62
Branches 769 759 -10
===========================================
- Hits 4548 4409 -139
- Misses 836 915 +79
+ Partials 197 195 -2
Continue to review full report at Codecov.
|
* Giving a jupyterconsole option for starting POCS * Fixing weather * Fixing ports
* Missing requirement for `convert` on cli. * Put weather plots in main images directory
…s, such as how to run easily
| @@ -125,8 +125,7 @@ are some helper scripts to make this easier (from [here](https://cloud.google.co | |||
|
|
|||
| ``` | |||
| gcloud components install docker-credential-gcr | |||
| docker-credential-gcr configure-docker | |||
| docker-credential-gcr gcr-login | |||
| gcloud auth configure-docker | |||
| ``` | |||
There was a problem hiding this comment.
one thing to note here is that you can also use an IAM service account and login with docker directly if there's no other need for having gcloud installed on the devices. https://cloud.google.com/container-registry/docs/advanced-authentication basically you create a service account and grant it read only permissions to the GCR storage bucket for the project and then they can login with docker cli directly using the service account json key like this docker login -u _json_key -p "$(cat keyfile.json)" https://us.gcr.io (assuming the key is in ./keyfile.json)
There was a problem hiding this comment.
Thanks @programatt, sorry for going so long. I've spent some time deep in Docker land in the last few weeks and think it will work well, thanks for all the tips.
We do use gcloud and gsutil pretty extensively and rely on a GCP network connection, so we do need them to authenticate at some point. We are using service account keys in a number of places and they work well.
I think I know have it so the images are public so anyone should be able to pull them, which is what I'm going for.
There was a problem hiding this comment.
@wtgee awesome! Glad to hear it seems to be going well for this project.
* FITS solving uses pipe * Removing unused packages
* Install arduino-cli. * Split into separate images.
* udev added for gphoto2 detection
Missing line continuation
uPDATE FILE NAME
Upate file name
|
Closing in favor of #859 |
This presents a number of changes for building and running POCS and friends
in a docker environment. As of this PR, starting the environment (via docker-compose)
will launch four separate services (container images):
As of now, it will also map the following directories to the host:
It also appropriately sets up the internal network that allows the services to communicate
with each other over zeromq and exposes port 8080 on the host machine for reaching
PAWS.
The environment can be started with a helper script:
As of now there is one additonal option that can be passed (
jupyterlab) whichwill start a jupyterlab environment within the running pocs service, exposing port
8888 on the host for access.
The POCS repo is installed in development mode (i.e.
pip install -e .) so that changesto the
$POCSdir on the host will be reflected in the running service.To do still:
pull images, etc.
Note: this runs all of the docker images in
privilegedmode, meaning they have access to the host.Also:
pocs-baseimage that contains the POCS repositoryinstalled on an ubuntu 18.04 base image. This includes an install of all the
dependencies, including astrometry.net. Some of the apt-packages have been
cleaned up or changed as part of this PR.
$POCS/scripts/docker/run_pocs.sh) still needs some work.$PANUSERis root as everything is run inside the images.Closes #734
There is a coming PR for https://github.com/panoptes/PAWS that this PR relies on.