Skip to content

Commit

Permalink
Docker into develop 02 (#951)
Browse files Browse the repository at this point in the history
* Dockerize

This will merge all the docker components into a new branch in the main repo. As part of milestone 0.7 these items will be cleaned up as a step towards making the docker version the default (i.e. `develop`) version for milestone 0.8.

This is being merged now because there are too many changes within one PR and it will be better to have new PRs that branch off this `docker` branch. This will also allow for installs on new units from the `panoptes` user rather than my personal branches.

* Docker branch upstream merge - WIP (#911)

Keeping `docker` in line with `develop`.

* Update env_file

Adding local user id

* Fixing power entry (#928)

* Minor fixes

* * Excplit teardown of camera for simulator_sdk so we can remove from assigned list. Note that this should ideally be happening by the `__del__` of the `AbstractSDCCamera` but is not getting called.

* Fixtures scopes at function level. Not sure what this will break.

* Adding empty `obstructions` for location config.

* Adding empty `obstructions` for location config.

* Try to download the IERS table before starting travis test

* Remove mongo

* Download IERS

* Core
* Explicitly cast free_space boolean

Camera
* Pass the config_port to the subcomponents
* Allow for a default string representation (needed to help debug object before it has fully initialized)

Debug logging

* * Default str representation for camera subcomponents that haven't been fully initialized.
* Logging statements for debug (will be removed).

* test_pocs

* Updating all the POCSTIME to be more current.
* Lots of debugging for now (I'm about to switch to desktop and want to keep these in)

* Ensure seconds quantity for exptime

* Test POCS

* Update POCSTIMEs and targets.
* Make the used observation a fixture.

* Small log change

* Ensure a float

* Testing should default to testing pointing

* Removing PR debugging.

* Removing PR debugging.

* Changes from #927 (@zacharyt20)

* Changes from #934 (@zacharyt20)

* Ignore vscode project files

* Coverage tests

* Coverage tests (and a few `no cover` cheats)

* PanBase cleanup

* Config server always returns a `None` so skip exception checking here.
* Coverage tests

* Move social messaging test to panoptes-utils

* Updates to linking pretty image to match panoptes-utils

* Make pretty in pocs shell matched to `panoptes-utils`

* Adding `responses` library.

* Removing references to mongo

* Better dockerignore

* Small changes to logging statements for consistency and use with new logger.

* Updating requirements

* Use new version of `panoptes-utils`

* Adding `responses` to main required section. `pip-tools` can't handle
the `extras_require` section so nothing from those can make their way
into `requirements.txt`, which is a bit frustrating.

* Requirements changes
* Updating pyzmq to latest
* Removing tweepy

* Removing `dateparser` and `pyzmq`

* Removing `photutils` and `python-dateutil`.
Other cleanup to setup.py

* Fixing calls to the root logger

* Getting rid of logger calls.

* Fixing caplog for pytest with new logging.

* **BREAKING CHANGE**

Changing `get_root_logger` to the more intuitive `get_logger`

* Logger Fixes

* Switches to `get_logger`
* Better logger config
* Remove old log config file

* Fixing focuser calls to mask_saturated for updates panoptes-utils

* Don't pass logger to write_fits

* Adding custom formatter for console logger so there are even widths for columns

* Singleton console formatter so column widths are retained across imports

* Change some log levels to trace for setup items in testing

* * Handle missing thumbnail creation.
* Smarter wait for cooling cameras so tests don't take as long.
* Better log format for cooling cameras.

* Logger:

* Better singleton
* Better work with pytest
* Moved more thngs to trace level

Filterwheel:

* Increased tolerances on some tests for some reason. 😕

* Breaking some logger coverage for now

* pep8
    * Ignore a few minor violations with new pycodestyle
    * fix some errors

* Requirements:

* Adding all the modules to the `required` section in `setup.py`. This means that `pip-tools` will generate a full and complete `requirements.txt` but is otherwise annoying.
    * Helps to fix some depedency issues in the docker image.

* Removing logger coverage for now

* Allow camera time to properly cool

* Some some debug messages when camera not ready.

* Increase default temperature_tolerance for ZWO cameras (#959)

* override default value

* Updating to `panoptes-utils==0.2.2` to fix testing issues.

* Fix paths to testing script

* Change GH action name

* Update version number.

* Updates to readme

* Auto fix format errors on changelog

* Updates to README and CHANGELOG

* Increase log retention policy

* Changing relevant `super()` calls.

* Remove unused script

* Changelog updates

* Move pdf manuals to resources directory

* Build documentation correctly

* Update readthedocs config

* Split the AAG weather docker containers to separate compose file.

* Fix `super` calls

* Update the install script

* Use `panoptes-exp`
* Remove the branch option
* Only link env file if one doesn't exist
* Newer version of `docker-compose`
* List of docker images updated

* Changing log levels (lots of output)

* * Timeout the GitHub Actions at 60 minutes (default is 6 hours!)
* Fix missing param

* * Don't unpark the mount in the `ready` state but wait until `slewing`.
* Better machine logic logging
* Small cleanup

* Generate all new documentation.

* More fixes for documentation

* Documentation fixes

* Fix tests with regard to parking change

* Change level colors for logger

* Fix url in setup
Fix some docs

* Trying to import cooling_enabled test.

* Don't capture log output

* Small test fixes

* Revert the travis test option

* Slight delay to killing of server so next one can start

* * Update `panoptes-utils`

* Reduce test verbose output

* f-string for vesion (why not?)

* Add back test verbosity (didn't help with GH Actions output)

* Die on first error

* Save log files in github if tests fail.

* Use local path for artifacts.

* Fix GH Actions path

* More trying to fix GH Action path

* Moving coverage upload to gh actions.

* Trying to figure out GHA artifacts

* Apparently need a name

* Fix pipes

* Add back the `failure`

* More GHA test

* GHA tests

* Docker container not a tty

* Try local logs

* More and more and more testing of GHA

* Remove test for testing tests

* Better name for job step

* Change step name for GHA

* Travis consistently fails on this test so attempting to get some debug info.

* Combine covergae in GHA

* z

* Make `coverage` command available.

* Always create log file artifact, only do coverage on success

* Camera tests will always wait for camera to be ready.

* Increase time allowed to get ready and log

* Identify which camera

* Try different wait for camers

* Adding ridiculous debugging.

* Try to debug random temperature spikes

* Better debug logging

* Decrease random temp variance on simulator.

* Just trace the temp instead of debug

* Coverage is combined at the time it is run.

* Lower temp var for simulator

* Set trace log level on CI testing.

* List files to see coverage

* Get some log output

* Use the logger from the camera.

* Compress console log file

* Also list hidden files on success (looking for travis file)

* Add some echoes

* Removing teardown

* Try explicit delete

* Trying to get more debug info on stalling tests

* Only log if have assigned cameras

* Updating panoptes-utils

* Send coverage reports from inside the testing file.

* Correct expression

* I thought I had the `or` working earlier but don't want to spend any more time.

* Coverage

* Explicit GH actions tests to find coverage

* * Opinionated logger forces handlers.
* No `SystemExit` on missig config items.
* Smarter filterwheel timeouts.
* Cleanup `camera` fixture so it skips hardware tests better.
* Adding a `testing` level for logger for differentiating between test output.

* Minor cleanup

* Comment about weird code for removing `simulator_sdk`

* Cheating with not testing caplog

* * Force logger handler singletons by id again. Seems like `loguru` should have a mechanism to track by name.

* Fix coverage artifact

* * Remove ci env and list before upload.

* Don't upload coverage file.

* Try to explictyly upload coverage file

* Show coverage report before uploading.

* Coverage report from test script.

* Test and coverage cleanup.

* * Add coverage upload action to pythontest GHA.
* Consolidate setup files into `setup.cfg`. Also rearrage `setup.cfg`.
    * Remove `.pycodestyle.cfg`
    * Remove `.coveragerc`

* Add to changelog.

* Better step names for GHA

* Fix codestyle test to point to config.

* Rename codecov file to be their selfish top-level name.

* More in changelog.

* * Default the install to the `panoptes` user intead of `$USER`.
* Don't get PAWS repo.

* Remove simple weather script reader and `pandas` dependency (moved to branch in `aag-weather`)

* Consistent script names

* Simplify env file

* Add `$PANUSER` to env file.

* * Revert back to installing with the `$USER`, which makes esnse for the host system.
* Remove `$LOCAL_USER_ID` from env file.

* Install script
* Don't install PAWS.
* Don't use env file (for now?).

Env file
* Move out of `docker` to root of project.

Config file cleanup
* Rearrange.
* Remove weather items.
* Remove social items.

* Install script

* Prompt for PANDIR.

* Remove stale comment

* Clarify misleading env var value

* Remove stale comment

* Clean up coverage and test files

* Use `PanLogger` from the utils.

* Manually addy the panoptes-utils github to requirements for now.

* Fix coverage pointing location.

* Add coverage variables.

* Make sure to add `latest` image.

* Update panoptes-utils

* Update labels and build for docker

* Small cleanups.

* Fixing requirements.

* * Supporting tagged Dockerfile so we can create a `develop` version (TODO).
* Adjusting test so that it builds docker image. Hopefully won't be too slow.

* Switch to panoptes user.

* Update Dockerfile.

* Don't link conf_files. Need a better solution.

* Bringing `PanLogger` (dynamic padding and handler info) from `panoptes-utils`.

* Cleanup of files and ignore files.

* Try to run the GHA tests with GHA user.

* Updates to test and config to try and make more like working panoptes-utils.

* Bumping panoptes-utils requirements

* Removing unnecessary comments.

* Updating test scripts for automated testing inside a container, either local or on CI service.

* Allow the CI service to run as its local user.

* Try GHA testing without local user id.

* Trying to solve log file permissions problem on GHA.

* Can't take time to fix this test as it might be removed.

* * Adding setuptools-scm
* Updating requirements.txt

* Ignore generated version file

* Small docker changes

* PyScaffold migration commit.

* Custom edits to the pyscaffold merge.

* **Breaking**

* Changing pocs namespace to fit within panoptes.

* Fixing tests.

* More big chnages for pyscaffold. Getting the docs and the tests working.

* Add static images

* More documentation updates and fixes.

Some filterwheel tolerance fixes.

* * Changing logging namespace.
* Updating contributing guide and other docs.

* * Cleaing up docker.
* Changing logger namespace back to original.

* * Add `is_sleeping` property.
* Move `parse_lines` into function as a camera utility.
* Using panoptes-utils serializers.

* * Fixing `parse_config` for gphoto2
* Added `is_sleeping` for pocs.
* Dockerfile install local via pip.

* * Adding items from #970.
* Using `panoptes.utils.time.wait_for_events`.
* Install instructions in the README.

* Readme cleanup.

* Readme cleanup.

* Test cleanup.

* * Making proper abstractmethods.
* Documentation updates where found.
* Many log and f-string fixes.
* `pocs.config_port` property available publicly.
* horizon check for state happens directly in `run`.
* `sleep` renamed to `wait`.
* dome status changed to dict.

* Changelog, quick test fixes.

* Fix bad import.

* Fix bad import.

* Install script

* Don't start time service on linux
* Don't write existing env vars to zshrc
* Don't fetch origin if repo already exists
* Always pull docker images
* Don't use sudo for pulling images

* Docker updates to use testing.

* Fixing GHA .

* Update contributing guide.

* Changelog updates.

* Changelog updates.

* Adding an update warning.
  • Loading branch information
wtgee authored Jun 1, 2020
1 parent 57a9938 commit 14400bd
Show file tree
Hide file tree
Showing 299 changed files with 7,057 additions and 20,788 deletions.
20 changes: 0 additions & 20 deletions .coveragerc

This file was deleted.

16 changes: 16 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
!.git
docs/*
.eggs
.idea
.venv
venv
*.egg-info
.github

*.md
!README*.md

*.log
*.pdf
**/*.pyc
**/__pycache__
75 changes: 46 additions & 29 deletions .github/workflows/pythontest.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Test Python Package
name: Test Python

on: [push, pull_request]
jobs:
Expand All @@ -8,37 +8,54 @@ jobs:
matrix:
python-version: [3.7]
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Lint with flake8
run: |
pip install flake8
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=100 --statistics
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Lint with flake8
run: |
pip install flake8
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=100 --statistics
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7]
steps:
- uses: actions/checkout@v2
- name: Fetch all history for all tags and branches for versioneer
run: git fetch --prune --unshallow
- name: Pull pocs image
run: |
docker pull gcr.io/panoptes-exp/pocs:latest
- name: Test with pytest in pocs container
run: |
ci_env=`bash <(curl -s https://codecov.io/env)`
docker run -i \
$ci_env \
-e LOCAL_USER_ID=$(id -u) \
-e TRAVIS=true \
-v $(pwd):/var/panoptes/POCS \
gcr.io/panoptes-exp/pocs:latest \
scripts/testing/run-tests.sh
- name: Checkout code
uses: actions/checkout@v2
- name: Fetch all history for all tags and branches
run: git fetch --prune --unshallow
- name: Build pocs image
run: |
docker build -t pocs:testing -f docker/latest.Dockerfile .
- name: Test with pytest in pocs container
run: |
mkdir -p coverage_dir && chmod 777 coverage_dir
ci_env=`bash <(curl -s https://codecov.io/env)`
docker run -i \
$ci_env \
-e REPORT_FILE="/tmp/coverage/coverage.xml" \
--network "host" \
-v $PWD:/var/panoptes/logs \
-v $PWD/coverage_dir:/tmp/coverage \
pocs:testing \
scripts/testing/run-tests.sh
- name: Upload coverage report to codecov.io
uses: codecov/codecov-action@v1
if: success()
with:
name: codecov-upload
file: coverage_dir/coverage.xml
fail_ci_if_error: true
- name: Create log file artifact
uses: actions/upload-artifact@v1
if: always()
with:
name: log-files
path: panoptes-testing.log
116 changes: 39 additions & 77 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,89 +1,51 @@
# PANOPTES specific files
conf_files/*_local.yaml
examples/notebooks/*.fits
examples/notebooks/*.jpeg

# Development support
sftp-config.json

# emacs backups
# Temporary and binary files
*~
\#*\#

# TeX products
*.aux
*.log
*.pdf
*.toc

# Compiled files
*.py[co]
*.a
*.o
*.py[cod]
*.so
__pycache__

# Ignore .c files by default to avoid including generated code. If you want to
# add a non-generated .c extension, use `git add -f filename.c`.
*.c

# Other generated files
_build
*/cython_version.py
htmlcov
.coverage
.coverage.*
MANIFEST

# Sphinx
docs/api
docs/_build
docs/_static
*.cfg
!.isort.cfg
!setup.cfg
*.orig
*.log
*.pot
__pycache__/*
.cache/*
.*.swp
*/.ipynb_checkpoints/*
.DS_Store

# Eclipse editor project files
# Project files
.ropeproject
.project
.pydevproject
.settings

# Pycharm editor project files
.idea
tags

# Packages/installer info
# Package files
*.egg
*.eggs
*.cache
*.egg-info
dist
build
eggs
parts
var
sdist
develop-eggs
*.eggs/
.installed.cfg
distribute-*.tar.gz

# Other
.*.swp

# Mac OSX
.DS_Store

# Eclipse project files
.settings
.project
.pydevproject

# Ignore IPython notebook (Jupyter) checkpoints.
.ipynb_checkpoints
notebooks/*

# Ignore link to weather_plots in web/static/
weather_plots
*.egg-info

# Ignore pytest.ini file in root of project. Especially useful
# on a unit to skip tests that interact with hardware.
/pytest.ini
# Unittest and coverage
htmlcov/*
.coverage
.tox
junit.xml
coverage.xml
.pytest_cache/

# Build and docs folder/files
build/*
dist/*
sdist/*
docs/api/*
docs/_rst/*
docs/_build/*
cover/*
MANIFEST

# Ignore pytest's cache of data across tests.
/.pytest_cache
# Per-project virtualenvs
.venv*/
**/.ipynb_checkpoints/**
16 changes: 0 additions & 16 deletions .pycodestyle.cfg

This file was deleted.

22 changes: 15 additions & 7 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,20 @@
# .readthedocs.yml
build:
image: latest
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

formats:
- htmlzip
# Required
version: 2

requirements_file: docs/requirements.txt
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/conf.py

formats: all

python:
version: 3.6
setup_py_install: true
version: 3.7
install:
- requirements: docs/requirements.txt
- method: pip
path: .
system_packages: true
Loading

0 comments on commit 14400bd

Please sign in to comment.