Feature/circleci fv3net build push (#192)

* Add .editorconfig

* Add HISTORY.rst

* Add RELEASE.rst

* Update manifest, probably incomplete

* Add version to fv3net init

* Add bumpversion to environment.yml

* Fix bumpversion path to __init__

* Get important image making files from noah

* Add noahs environment scripts

* Temporary testing with different env setups

* Refactore online_modules to fv3net

* black

* Create us.gcr.io/vcm-ml/prognostic_run:v0.1.0

Builds a docker image with fv3net, vcm, and fv3gfs-python installed.

* Refactor us.gcr.io/vcm-ml/fv3net image build code

* Add build_images makefile target

* Add __version__ to fv3net init

* update prognostic_run_diags configuration

* black

* update readme

* Fix table in README

* fix yaml bug

* pin pandas version to 1.0.1

* Revert "Get important image making files from noah"

This reverts commit ebb0d4c.

* Revert "Add noahs environment scripts"

This reverts commit 956936f.

* Fix environment.yml from merge

* Add conda update to CircleCI file

* Add caching to the CircleCI for fv3net deps

* Add cache saving

* Fix ci config

* Move temp pkg directory creation

* Conditional on tmp package directory

* Install fv3net packages

* Fix cached directory move

* Move conda cache explicitly

* Switch conda cache move to copy op

* Add debug ls

* Switch to force move

* Add mv -f for conda cache

* Add make target for creating CircleCI image

* Back to normal mv command after clearing conda cache in docker image

* Clear CircleCI cache by bumping v1->v2

* Remove glob from save_cache

* Add in conda clean, remove existence check before creating tmp pkg directories

* Invalidate cache one more time v2 -> v3

* Fix cache restore mv commands

* Add cache for the leftover conda packages

* Test without caching

* Caching wasn't worth it.  Trying env cache

* Adjust build to include full list of packages and update environment.yml for speed

* Update CI cache to include more explicit representation of installed deps

* Fix import errors

* Remove temporary files

* Remove fv3net packages from the cached files

* Add build push for circleci on versioned release

* Fix make install

* Try gcp-gcr auth

* Test build only

* Try decoded creds

* Add compute zone

* Output ci branch/tag vars

* Test no build_default on branch push

* Undo all branch ignore

* Tagged release on feature branch

* change tag filter

* Try different credentials for CI bucket access

* Export env var to BASH_ENV for gcr-auth

* Switch to decoded credentials again

* Try push without gcr-auth command

* Fix credential env variable

* Switch back to gcr-auth orb

* Update release procedures

* Update build filters back to master only

* Add bumpversion entry for makefile

* Fix ci config error

* Revert "Merge branch 'refactor/fv3net-online-modules' into feature/versioned-fv3net"

This reverts commit 8126de2, reversing
changes made to 2132dfc.

* Add a note about the .install_build_env_deps file to the README

* Final cleanup of files

* Match fv3config submodule to master

* Update documentation about contributing, history, and releases

* Small text fix for RELEASE.rst

* Fix RELEASE rst heading syntax

* Add release files for vcm

* Update files to adjust with bumpversion

* Update release procedure for fv3net

* Fix bumpversion file path specifcation

* Testfix for bumpversion setup config search/replacement strings

* Test string style change in setup.cfg

* Final versioning string fix

* Clarify bumpversion usage

* Add prognostic run version push to circleci

* Add back in section header separator for COMMANDS

* Add pip wheel dependencies for make prognostic image

* Test CI job for docker pull

* Try prognostic image build

* Explicity use of working dir for pip wheel

* Adjust docker ignore for CI directory

* Try adjust docker build call

* Add whl files into docker/prognostic_run folder instead?

* Try deleting everything in dockerignore

* Undo dockerignore changes add ls debug statement

* Fix make ls command for debug

* Remove pip wheel install

* Upgrade setuptools as well

* Use python 3

* Fix python version

* Switch to python 3.7.0

* Switch to miniconda3-3.7.0

* Update python using conda

* Try with unpulled fv3gfs version

* Undo testing changes to makefile and dockerfile for prognostic

* Remove testing jobs from CircleCI

* Allow wheel dir specification and add pyenv switch for docker

* Add pip update to make wheel work in python 2.7

* Add in dockerfile features from another branch

Co-authored-by: W. Andre Perkins <[email protected]>

* Update CircleCI after dockerfile build updates

* Use push_images

* Add submodule init/update

* Fix indent on submodule ci step

* Fix config.yml for image push version->VERSION

* Final clean up of leftover testing changes

* Remove unused code to address review comments

* add script to run tests in docker image after build

* Reset environment.yml

Co-authored-by: Noah D. Brenowitz <[email protected]>

.circleci/config.yml

@@ -1,6 +1,8 @@
 version: 2.1
+orbs:
+  gcp-gcr: circleci/[email protected]
 jobs:
-  build:    
+  pytest_default:    
     docker:
       - image: us.gcr.io/vcm-ml/circleci-miniconda-gfortran:latest
         auth:
@@ -9,26 +11,112 @@ jobs:
         environment:
           GOOGLE_APPLICATION_CREDENTIALS=key.json
     steps:
-    - checkout
-    - run: | 
-        echo $ENCODED_GOOGLE_CREDENTIALS | \
-        base64 --decode > $GOOGLE_APPLICATION_CREDENTIALS
-    - run: conda install make
-    - run: make update_submodules
-    - run: make create_environment
-    - run: | 
-        source activate fv3net
-        pytest -m "not regression" --mpl --mpl-baseline-path=tests/baseline_images
+      - checkout
+      - run:
+          name: "Save GCS for tests."
+          command: |
+            echo $ENCODED_GOOGLE_CREDENTIALS | \
+            base64 --decode > $GOOGLE_APPLICATION_CREDENTIALS
+      - run: make update_submodules
+      - run:
+          name: "Concatenate package dependency files"
+          command: cat environment.yml .circleci/.installed_build_env_deps > combined_deps.txt
+      - restore_cache:
+          keys:
+            - v2-fv3net-env-{{ checksum "combined_deps.txt" }}
+      - run:
+          name: "Install fv3net dependencies"
+          command: |
+            make install_deps
+            cat .circleci/.installed_build_env_deps
+      - save_cache:
+          key: v2-fv3net-env-{{ checksum "combined_deps.txt" }}
+          paths:
+            - /opt/conda/envs/fv3net
+      - run:
+          name: "Install fv3net packages"
+          command: make install_local_packages
+      - run: | 
+          source activate fv3net
+          pytest -m "not regression" --mpl --mpl-baseline-path=tests/baseline_images
+
+  build_default:
+    machine:
+      docker_layer_caching: true
+    environment:
+      GOOGLE_PROJECT_ID: vcm-ml
+      GOOGLE_COMPUTE_ZONE: us-central1
+      GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
+    steps:
+      - gcp-gcr/gcr-auth:
+          gcloud-service-key: DECODED_GOOGLE_CREDENTIALS
+      - checkout
+      - run: sudo apt-get install make
+      - run:
+          name: "Pull submodules"
+          command: |
+            git submodule update --init --recursive
+      - run:
+          name: Build fv3net images 
+          command: |
+            if [[ -z "$CIRCLE_TAG" ]]
+            then
+                use_version="latest"
+            else
+                use_version=$CIRCLE_TAG
+            fi
+            make build_images VERSION=$use_version
+      - run:
+          name: Run tests in fv3net image
+          command: |
+              echo $DECODED_GOOGLE_CREDENTIALS > $GOOGLE_APPLICATION_CREDENTIALS
+              bash docker/fv3net/run_tests_in_image.sh $CIRCLE_TAG
+      - run:
+          name: Push fv3net images
+          command: |
+            echo $CIRCLE_BRANCH
+            echo $CIRCLE_TAG
+            if [[ "$CIRCLE_BRANCH" == "master" ]]
+            then
+                echo "pushing untagged images as 'latest'"
+                make push_images VERSION=latest
+            fi
+            if [[ ! -z "$CIRCLE_TAG" ]]
+            then
+                echo "pushing tagged images $CIRCLE_TAG"
+                make push_images VERSION=$CIRCLE_TAG
+            fi
+  
   lint:
     docker:
     - image: circleci/python:3.7
     steps:
     - checkout
     - run: sudo pip install black==19.10b0 flake8
     - run: make lint
+
+
 workflows:
   version: 2
   test_and_lint:
     jobs:
       - lint
-      - build
+      - pytest_default
+  build_and_push:
+    jobs:
+      - pytest_default:
+          filters:
+            tags:
+              only: /^v.*/
+            branches:
+              only: master
+      - build_default:
+          requires: 
+            - pytest_default
+          filters:
+            tags:
+              only: /^v.*/
+            branches:
+              only: master
+
+

.circleci/dockerfile

@@ -7,6 +7,9 @@ RUN apt-get install -y curl && \
     dpkg -i linux-libc-dev.deb && \
     rm -f linux-libc-dev.deb && \
     apt-get install -y gfortran && \
-    apt-get clean
+    apt-get install -y make && \
+    apt-get clean && \
+    conda update -n base conda && \
+    conda clean -i
 
 CMD [ "/bin/bash" ]

.editorconfig

@@ -0,0 +1,21 @@
+# http://editorconfig.org
+
+root = true
+
+[*]
+indent_style = space
+indent_size = 4
+trim_trailing_whitespace = true
+insert_final_newline = true
+charset = utf-8
+end_of_line = lf
+
+[*.bat]
+indent_style = tab
+end_of_line = crlf
+
+[LICENSE]
+insert_final_newline = false
+
+[Makefile]
+indent_style = tab

.environment-scripts/build_environment.sh

@@ -3,5 +3,6 @@
 CONDA_ENV=$1
 CONDA_BASE=$(conda info --base)
 
-conda env create -n $CONDA_ENV -f environment.yml  2> /dev/null || \
+conda env create -n $CONDA_ENV -f environment.yml  2> /dev/null && \
+	conda list -n $CONDA_ENV > .circleci/.installed_build_env_deps || \
 	echo "Conda env already exists proceeding to VCM package installation"

CONTRIBUTING.md

@@ -1,5 +1,8 @@
 # Contributions Guides and Standards
 
+Please record all changes and updates in [HISTORY.rst](./HISTORY.rst) under the 
+upcoming section.  It is especially important to log changes that break backwards 
+compatibility so we can appropriately adjust the versioning.
 
 ## fv3net
 

HISTORY.rst

@@ -4,8 +4,10 @@ History
 
 Current
 -------
+* Added bump2version for automated versioning of `fv3net` resources 
+* Add CircleCI build/push capabilities for `fv3net` images
+
 
 0.1.0 (2020-03-20)
 ------------------
-
 * First release of fv3net

MANIFEST.in

@@ -1 +1,8 @@
-include fv3net/regression/sklearn/default_norm_outputs.dat
+
+include HISTORY.rst
+include LICENSE
+include README.rst
+
+recursive-include tests *
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]

Makefile

@@ -3,7 +3,8 @@
 #################################################################################
 # GLOBALS                                                                       #
 #################################################################################
-VERSION = v0.1.1
+
+VERSION ?= v0.1.0
 ENVIRONMENT_SCRIPTS = .environment-scripts
 PROJECT_DIR := $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
 BUCKET = [OPTIONAL] your-bucket-for-syncing-data (do not include 's3://')
@@ -23,22 +24,22 @@ endif
 #################################################################################
 # COMMANDS                                                                      #
 #################################################################################
-.PHONY: wheels build_images push_image
-wheels:
-	pip wheel --no-deps .
-	pip wheel --no-deps external/vcm
+
+.PHONY: build_images push_image
 
 # pattern rule for building docker images
 build_image_%:
-	docker build -f docker/$*/Dockerfile . -t us.gcr.io/vcm-ml/$*:$(VERSION)
+	docker build . -f docker/$*/Dockerfile -t us.gcr.io/vcm-ml/$*:$(VERSION)
 
-build_image_prognostic_run: wheels
+enter_%:
+	docker run -ti -w /fv3net -v $(shell pwd):/fv3net us.gcr.io/vcm-ml/$*:$(VERSION) bash
 
 build_images: build_image_fv3net build_image_prognostic_run
 
-push_image:
-	docker push us.gcr.io/vcm-ml/fv3net:$(VERSION)
-	docker push us.gcr.io/vcm-ml/prognostic_run:$(VERSION)
+push_images: push_image_prognostic_run push_image_fv3net
+
+push_image_%:
+	docker push us.gcr.io/vcm-ml/$*:$(VERSION)
 
 enter: build_image
 	docker run -it -v $(shell pwd):/code \
@@ -48,6 +49,9 @@ enter: build_image
 #		-e GOOGLE_APPLICATION_CREDENTIALS=/google_creds.json \
 #		-v $(HOME)/.config/gcloud/application_default_credentials.json:/google_creds.json \
 
+build_ci_image:
+	docker build -t us.gcr.io/vcm-ml/circleci-miniconda-gfortran:latest - < .circleci/dockerfile
+
 
 ## Make Dataset
 .PHONY: data update_submodules create_environment overwrite_baseline_images

README.md

@@ -66,6 +66,12 @@ succesfully, the fv3net environment can be activated with
 
     conda activate fv3net
 
+The `build_environment.sh` script also outputs a list of all installed dependencies
+with their version under `.circleci/.installed_build_env_deps`.  This file is used
+along with `environment.yml` as a key for caching the `fv3net` dependencies.  Whenver
+`make create_environment` or the build script is run, this file will be updated
+and committed to keep track of versions over time.
+
 # Deploying cloud data pipelines
 
 The main data processing pipelines for this project currently utilize Google Cloud
@@ -208,4 +214,14 @@ If it does not pass, than it can be autoformatted using
 
     make reformat
 
-<p><small>Project based on the <a target="_blank" href="https://drivendata.github.io/cookiecutter-data-science/">cookiecutter data science project template</a>. #cookiecutterdatascience</small></p>
+# How to contribute to fv3net
+
+Please see the [contribution guide.](./CONTRIBUTING.md)
+
+# How to get updates and releases
+
+For details on what's included or upcoming for a release, please see the [HISTORY.rst](./HISTORY.rst) document.
+
+For instructions on preparing a release, please read [RELEASE.rst](./RELEASE.rst).
+
+<p><small>Project based on the <a target="_blank" href="https://drivendata.github.io/cookiecutter-data-science/">cookiecutter data science project template</a>. #cookiecutterdatascience</small></p>

RELEASE.rst

@@ -0,0 +1,24 @@
+====================
+Release Instructions
+====================
+
+Release naming guidelines
+-------------------------
+
+If `x.y.z` is the version, bump `y` (minor) on new features or breaking changes, and `z` on smaller changes.
+
+Release steps
+-------------
+
+1. Prepare master branch for release (make sure all PRs are merged and tests pass).
+
+2. Create a branch for the version bump and run `bumpversion <part>` where part
+   is the section of the version to bump. I.e., for x.y.z, the corresponding "part"
+   would be "major" for x, "minor" for y, and "patch" for z.
+   This will create a new commit, having updated all version references to the new, 
+   higher version.
+
+3. Push the bump branch to Github to open a PR. 
+
+3. Merge this branch to master on Github and create a release version consistent 
+   with the version bump.

docker/fv3net/Dockerfile

@@ -10,6 +10,7 @@ USER root
 RUN apt-get update && apt-get install -y gfortran
 ADD environment.yml  $FV3NET/
 ADD .environment-scripts $ENVIRONMENT_SCRIPTS
+ADD .circleci $FV3NET/.circleci
 RUN fix-permissions $FV3NET
 WORKDIR $FV3NET
 

docker/fv3net/run_tests_in_image.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+FV3NET_IMAGE="us.gcr.io/vcm-ml/fv3net"
+
+set -e
+
+if [[ -z $1 ]]; then
+    VERSION_TAG="latest"
+else
+    VERSION_TAG=$1
+fi
+
+if [[ -z $GOOGLE_APPLICATION_CREDENTIALS ]]; then
+    docker run -it $FV3NET_IMAGE:$VERSION_TAG py.test
+else
+    docker run -v $GOOGLE_APPLICATION_CREDENTIALS:/tmp/json.key \
+        --env GOOGLE_APPLICATION_CREDENTIALS=/tmp/json.key \
+        -it $FV3NET_IMAGE:$VERSION_TAG "py.test"
+fi

docker/prognostic_run/Dockerfile

@@ -3,6 +3,11 @@ FROM us.gcr.io/vcm-ml/fv3gfs-python:v0.3.1
 
 COPY docker/prognostic_run/requirements.txt /tmp/requirements.txt
 RUN pip3 install -r /tmp/requirements.txt
-COPY fv3net-0.1.0-py3-none-any.whl /wheels/fv3net-0.1.0-py3-none-any.whl
-COPY vcm-0.1.0-py3-none-any.whl /wheels/vcm-0.1.0-py3-none-any.whl
-RUN pip3 install --no-deps /wheels/fv3net-0.1.0-py3-none-any.whl  && pip3 install /wheels/vcm-0.1.0-py3-none-any.whl
+
+# cache external package installation
+COPY external/fv3config /fv3net/external/fv3config
+COPY external/vcm /fv3net/external/vcm
+RUN pip3 install -e /fv3net/external/vcm -e /fv3net/external/fv3config
+
+COPY . /fv3net
+RUN pip3 install --no-deps -e /fv3net

docker/prognostic_run/requirements.txt

@@ -3,4 +3,5 @@ dask
 joblib
 zarr
 scikit-image
+google-cloud-logging
 backoff

environment.yml

@@ -45,7 +45,9 @@ dependencies:
   - python=3.7
   - holoviews
   - bokeh=1.4
+  - backoff
   - pip:
     - gsutil
     - nc-time-axis>=1.2.0
+    - bump2version>=0.5.11
     - yq

external/vcm/HISTORY.rst

@@ -2,7 +2,10 @@
 History
 =======
 
-0.1.0 (2019-10-11)
-------------------
+Current
+-------
+
 
-* First release on PyPI.
+0.1.0 (2020-03-20)
+------------------
+First VCM release (w/ fv3net for now)

external/vcm/MANIFEST.in

@@ -3,8 +3,6 @@ include HISTORY.rst
 include LICENSE
 include README.rst
 
-recursive-include fv3config/data *
-
 recursive-include tests *
 recursive-exclude * __pycache__
 recursive-exclude * *.py[co]