Release covidcast-indicators 0.3.56 (#2049)

* doc: add logger examples to README and format * deployment + staging details * wrap lines * Update _template_python/INDICATOR_DEV_GUIDE.md Co-authored-by: David Weber <[email protected]> * Update _template_python/INDICATOR_DEV_GUIDE.md Co-authored-by: David Weber <[email protected]> * Update _template_python/INDICATOR_DEV_GUIDE.md Co-authored-by: David Weber <[email protected]> * Update INDICATOR_DEV_GUIDE.md with mysql syntax * Update INDICATOR_DEV_GUIDE.md reorder instruction on cron jobs * switch from input_file to input_dir (#1995) * adjust nssp max_age * add pyproject.toml * made changes based on suggestion * update ci config * separate new line Co-authored-by: Dmitry Shemetov <[email protected]> * fix: add explicit boto and moto dependencies to indicators * trying to get package data to show * installing dev package before testing in ci * adding dependency * move comment to right package * Update _delphi_utils_python/pyproject.toml Co-authored-by: Dmitry Shemetov <[email protected]> * Revert "Clean up delphi_utils " * refactoring date parsing to seperate file * first implimentation * small cleanup and adding missing data * changed wording and some logic * made test more explicit * add missing file * cleaning up tests * consolidating tests into one * fixed wording * consolidating test/fixture * clean up test * Update google_symptoms/delphi_google_symptoms/patch.py remove old code; not used anymore Co-authored-by: nmdefries <[email protected]> * clean up lint * remove teardown * cleaning and renaming * more explicit branching for regular run vs patching, renaming, cleaning * cleanup and rewrite logic * avoid linking keywords in PR template * Update _delphi_utils_python/README.md Co-authored-by: george <[email protected]> * Refactor Jenkins pipeline - Remove usage of parallel in the build and package stages - Retain usage of parallel in the deploy stages - Separate build and package into separate scripts to support build-only stages - Add a build-only stage for dev/feature branches * Add separate build and package scripts * Update script comments * Remove unused script * changes based on suggestion * standardizing output dir in test * changing back to session scope * more logging and update params * adding back conditional for num_export_days * test_run does not like fixtures * make deep copies of params fixtures * linting * test patch check if num_export_days is set or not * patch fn takes params as an arg, like run_module * list values added using actual day as an example * move pull date creation into pull_gs_data * don't modify num_export_days; define start/end date in terms of it and lag * refer to params by name; patch_flag -> custom_run flag * use test data based on recent actual indicator run * say where gold test data came from * not all date_utils tests need metadata * formatting * fn docs * lint and cleanup * adding more tests * removed extra lag and made test more robust * export_start_date is optional since the first available date is baked in * need to account for historical / current run for adding lag * adding validation script * fix typo * fixing typo in test * lint * sample run for validation * remove unused input Co-authored-by: nmdefries <[email protected]> * remove unused import * added progress chunk to limit logging (#2024) * added progress chunk to limit logging * lint * comment based on suggestion * add hhs geo level (#2036) * Add discussion of patching functionality to indicator manual (#2031) * basic patching info * custom_run * add hhs to geo section * de-emphasize R in contributing doc, and add links * ref/issue date intro * naming and force push standards * secret and params links * Clean up delphi_utils (take 2) (#2014) * add pyproject.toml * made changes based on suggestion * update ci config * separate new line Co-authored-by: Dmitry Shemetov <[email protected]> * fix: add explicit boto and moto dependencies to indicators * trying to get package data to show * installing dev package before testing in ci * adding dependency * move comment to right package * Update _delphi_utils_python/pyproject.toml Co-authored-by: Dmitry Shemetov <[email protected]> * more suggestion * adding pyproject.toml to template * Update changehc/Makefile * Update claims_hosp/Makefile * Update doctor_visits/Makefile * Update google_symptoms/Makefile * Update hhs_hosp/Makefile * Update nchs_mortality/Makefile * Update nssp/Makefile * Update nwss_wastewater/Makefile * Update quidel_covidtest/Makefile * Update sir_complainsalot/Makefile * fix: dependence bugs and cleanup * pandas[excel]<2 doesn't work, go back to xlrd and comment * add mock to changehc * add requests * remove setup.py from template * build: fix Jenkins build script --------- Co-authored-by: Amaris Sim <[email protected]> Co-authored-by: aysim319 <[email protected]> Co-authored-by: Dmitry Shemetov <[email protected]> * fix: build-indicators.sh looks for setup.py and pyproject.toml (#2048) * fix: build-indicators.sh looks for setup.py again Reverting a premature change. * fix: readability and future-proof * fix: more readability * chore: bump delphi_utils to 0.3.25 * chore: bump covidcast-indicators to 0.3.56 * [create-pull-request] automated change --------- Co-authored-by: Dmitry Shemetov <[email protected]> Co-authored-by: minhkhul <[email protected]> Co-authored-by: Nat DeFries <[email protected]> Co-authored-by: minhkhul <[email protected]> Co-authored-by: David Weber <[email protected]> Co-authored-by: george <[email protected]> Co-authored-by: Amaris Sim <[email protected]> Co-authored-by: aysim319 <[email protected]> Co-authored-by: Brian Clark <[email protected]> Co-authored-by: Delphi Deploy Bot <[email protected]> Co-authored-by: minhkhul <[email protected]>
cmu-delphi · Sep 12, 2024 · dd22ccd · dd22ccd
1 parent b338a09
commit dd22ccd
Show file tree

Hide file tree

Showing 59 changed files with 51,793 additions and 405 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.55
+current_version = 0.3.56
 commit = True
 message = chore: bump covidcast-indicators to {new_version}
 tag = False
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -12,7 +12,9 @@ The production branch is configured to automatically deploy to our production en
 
 * everything else
 
-All other branches are development branches. We don't enforce a naming policy.
+All other branches are development branches. We don't enforce a naming policy, but it is recommended to prefix all branches you create with your name, username, or initials (e.g. `username/branch-name`).
+
+We don't forbid force-pushing, but please keep to a minimum and be careful of using when modifying a branch at the same time as others.
 
 ## Issues
 
@@ -29,7 +31,7 @@ So, how does one go about developing a pipeline for a new data source?
 ### tl;dr
 
 1. Create your new indicator branch from `main`.
-2. Build it using the appropriate template, following the guidelines in the included README.md and REVIEW.md files.
+2. Build it using the [indicator template](https://github.com/cmu-delphi/covidcast-indicators/tree/main/_template_python), following the guidelines in the included README.md, REVIEW.md, and INDICATOR_DEV_GUIDE.md files.
 3. Make some stuff!
 4. When your stuff works, push your development branch to remote, and open a PR against `main` for review.
 5. Once your PR has been merged, consult with a platform engineer for the remaining production setup needs. They will create a deployment workflow for your indicator including any necessary production parameters. Production secrets are encrypted in the Ansible vault. This workflow will be tested in staging by admins, who will consult you about any problems they encounter.
@@ -50,7 +52,7 @@ git checkout -b dev-my-feature-branch
 
 ### Creating your indicator
 
-Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. If using Python, add the name of the directory to the list found in `jobs > build > strategy > matrix > packages` in `.github/workflows/python-ci.yml`, which will enable automated checks for your indicator when you make PRs. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach.
+Create a directory for your new indicator by making a copy of `_template_python`. (We also make a `_template_r` available, but R should be only used as a last resort, due to complications using it in production.) Add the name of the directory to the list found in `jobs > build > strategy > matrix > packages` in `.github/workflows/python-ci.yml`, which will enable automated checks for your indicator when you make PRs. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach.
 
 * Consult your peers with questions! :handshake:
 
@@ -62,7 +64,7 @@ Once you have something that runs locally and passes tests you set up your remot
 git push -u origin dev-my-feature-branch
 ```
 
-You can then draft public API documentation for people who would fetch this
+You can then draft [public API documentation](https://cmu-delphi.github.io/delphi-epidata/) for people who would fetch this
 data from the API. Public API documentation is kept in the delphi-epidata
 repository, and there is a [template Markdown
 file](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast-signals/_source-template.md)
@@ -104,7 +106,8 @@ We use a branch-based git workflow coupled with [Jenkins](https://www.jenkins.io
   * Package - Tar and gzip the built environment.
   * Deploy - Trigger an Ansible playbook to place the built package onto the runtime host, place any necessary production configuration, and adjust the runtime envirnemnt (if necessary).
 
-There are several additional Jenkins-specific files that will need to be created for each indicator, as well as some configuration additions to the runtime host. It will be important to pair with a platform engineer to prepare the necessary production environment needs, test the workflow, validate on production, and ultimately sign off on a production release.
+There are several additional Jenkins-specific files that will need to be created for each indicator, as well as some configuration additions to the runtime host.
+It will be important to pair with a platform engineer to prepare the necessary production environment needs, test the workflow, validate on production, and ultimately sign off on a production release.
 
 ### Preparing container images of indicators
 

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -6,5 +6,5 @@ Itemize code/test/documentation changes and files added/removed.
 - change1
 - change2
 
-### Fixes 
-- Fixes #(issue)
+### Associated Issue(s) 
+- Addresses #(issue)
diff --git a/.github/workflows/python-ci.yml b/.github/workflows/python-ci.yml
@@ -51,7 +51,7 @@ jobs:
         with:
           python-version: 3.8
           cache: "pip"
-          cache-dependency-path: "setup.py"
+          cache-dependency-path: "pyproject.toml"
       - name: Install testing dependencies
         run: |
           python -m pip install --upgrade pip

diff --git a/Jenkinsfile b/Jenkinsfile
@@ -10,7 +10,7 @@
    - TODO: #527 Get this list automatically from python-ci.yml at runtime.
  */
 
-def indicator_list = ["backfill_corrections", "changehc", "claims_hosp", "google_symptoms", "hhs_hosp", "nchs_mortality", "quidel_covidtest", "sir_complainsalot", "doctor_visits", "nwss_wastewater", "nssp"]
+def indicator_list = ['backfill_corrections', 'changehc', 'claims_hosp', 'google_symptoms', 'hhs_hosp', 'nchs_mortality', 'quidel_covidtest', 'sir_complainsalot', 'doctor_visits', 'nwss_wastewater', 'nssp']
 def build_package_main = [:]
 def build_package_prod = [:]
 def deploy_staging = [:]
@@ -19,39 +19,62 @@ def deploy_production = [:]
 pipeline {
     agent any
     stages {
-        stage('Build and Package main') {
+        stage('Build dev/feature branch') {
+            when  {
+                not {
+                    anyOf {
+                        branch 'main'
+                        branch 'prod'
+                    }
+                }
+            }
+            steps {
+                script {
+                    indicator_list.each { indicator ->
+                        stage("Build ${indicator}") {
+                            sh "jenkins/build-indicator.sh ${indicator}"
+                        }
+                    }
+                }
+            }
+        }
+        stage('Build and Package main branch') {
             when {
-                branch "main";
+                branch 'main'
             }
             steps {
                 script {
                     indicator_list.each { indicator ->
-                        build_package_main[indicator] = {
-                            sh "jenkins/build-and-package.sh ${indicator} main"
+                        stage("Build ${indicator}") {
+                            sh "jenkins/build-indicator.sh ${indicator}"
+                        }
+                        stage("Package ${indicator}") {
+                            sh "jenkins/package-indicator.sh ${indicator} main"
                         }
                     }
-                    parallel build_package_main
                 }
             }
         }
-        stage('Build and Package prod') {
+        stage('Build and Package prod branch') {
             when {
-                branch "prod";
+                branch 'prod'
             }
             steps {
                 script {
                     indicator_list.each { indicator ->
-                        build_package_prod[indicator] = {
-                            sh "jenkins/build-and-package.sh ${indicator} prod"
+                        stage("Build ${indicator}") {
+                            sh "jenkins/build-indicator.sh ${indicator}"
+                        }
+                        stage("Package ${indicator}") {
+                            sh "jenkins/package-indicator.sh ${indicator} prod"
                         }
                     }
-                    parallel build_package_prod
                 }
             }
         }
-        stage('Deploy staging') {
+        stage('Deploy main branch to staging env') {
             when {
-                branch "main";
+                branch 'main'
             }
             steps {
                 script {
@@ -64,9 +87,9 @@ pipeline {
                 }
             }
         }
-        stage('Deploy production') {
+        stage('Deploy prod branch to production env') {
             when {
-                branch "prod";
+                branch 'prod'
             }
             steps {
                 script {

diff --git a/_delphi_utils_python/.bumpversion.cfg b/_delphi_utils_python/.bumpversion.cfg
@@ -1,9 +1,9 @@
 [bumpversion]
-current_version = 0.3.24
+current_version = 0.3.25
 commit = True
 message = chore: bump delphi_utils to {new_version}
 tag = False
 
-[bumpversion:file:setup.py]
+[bumpversion:file:pyproject.toml]
 
 [bumpversion:file:delphi_utils/__init__.py]
diff --git a/_delphi_utils_python/DEVELOP.md b/_delphi_utils_python/DEVELOP.md
@@ -9,7 +9,7 @@ To install the module in your default version of Python, run the
 following from this directory:
 
 ```
-pip install .
+pip install -e '.[dev]'
 ```
 
 As described in each of the indicator code directories, you will want to install

diff --git a/_delphi_utils_python/Makefile b/_delphi_utils_python/Makefile
@@ -6,12 +6,12 @@ venv:
 install: venv
 	. env/bin/activate; \
 	pip install wheel ; \
-	pip install -e .
+	pip install -e '.[dev]'
 
 install-ci: venv
 	. env/bin/activate; \
-	pip install wheel ; \
-	pip install .
+	pip install 'build[virtualenv]' pylint pytest pydocstyle wheel twine ; \
+	pip install '.[dev]'
 
 lint:
 	. env/bin/activate; pylint delphi_utils --rcfile=../pyproject.toml
@@ -30,4 +30,5 @@ clean:
 
 release: lint test
 	. env/bin/activate ; \
-	python setup.py sdist bdist_wheel
+	pip install 'build[virtualenv]' ; \
+	python -m build --sdist --wheel
diff --git a/_delphi_utils_python/README.md b/_delphi_utils_python/README.md
@@ -17,5 +17,32 @@ Submodules:
 - `validator`: Data sanity checks and anomaly detection.
 
 
-Source code can be found here: 
-[https://github.com/cmu-delphi/covidcast-indicators/](https://github.com/cmu-delphi/covidcast-indicators/)
+Source code can be found here:
+[https://github.com/cmu-delphi/covidcast-indicators/](https://github.com/cmu-delphi/covidcast-indicators/)
+
+## Logger Usage
+
+Single-thread usage.
+
+```py
+from delphi_utils.logger import get_structured_logger
+
+logger = get_structured_logger('my_logger')
+logger.info('Hello, world!')
+```
+
+Multi-thread usage.
+
+```py
+from delphi_utils.logger import get_structured_logger, pool_and_threadedlogger
+
+def f(x, threaded_logger):
+    threaded_logger.info(f'x={x}')
+    return x*x
+
+logger = get_structured_logger('my_logger')
+logger.info('Hello, world!')
+with pool_and_threadedlogger(logger, n_cpu) as (pool, threaded_logger):
+    for i in range(10):
+        pool.apply_async(f, args=(i, threaded_logger))
+```
diff --git a/_delphi_utils_python/delphi_utils/__init__.py b/_delphi_utils_python/delphi_utils/__init__.py
@@ -15,4 +15,4 @@
 from .nancodes import Nans
 from .weekday import Weekday
 
-__version__ = "0.3.24"
+__version__ = "0.3.25"
diff --git a/_delphi_utils_python/delphi_utils/geomap.py b/_delphi_utils_python/delphi_utils/geomap.py
@@ -443,7 +443,7 @@ def add_population_column(
         ---------
         data: pd.DataFrame
             The dataframe with a FIPS code column.
-        geocode_type: {"fips", "zip"}
+        geocode_type:
             The type of the geocode contained in geocode_col.
         geocode_col: str, default None
             The name of the column containing the geocodes. If None, uses the geocode_type
@@ -671,8 +671,10 @@ def aggregate_by_weighted_sum(
             to a from_geo, e.g. "wastewater collection site").
         to_geo: str
             The column name of the geocode to aggregate to.
-        sensor: str
+        sensor_col: str
             The column name of the sensor to aggregate.
+        time_col: str
+            The column name of the timestamp to aggregate over.
         population_column: str
             The column name of the population to weight the sensor by.
 

diff --git a/_delphi_utils_python/delphi_utils/logger.py b/_delphi_utils_python/delphi_utils/logger.py
@@ -1,5 +1,7 @@
 """Structured logger utility for creating JSON logs.
 
+See the delphi_utils README.md for usage examples.
+
 The Delphi group uses two ~identical versions of this file.
 Try to keep them in sync with edits, for sanity.
   https://github.com/cmu-delphi/covidcast-indicators/blob/main/_delphi_utils_python/delphi_utils/logger.py
@@ -133,19 +135,17 @@ class LoggerThread():
     """
     A construct to use a logger from multiprocessing workers/jobs.
 
-    the bare structlog loggers are thread-safe but not multiprocessing-safe.
-    a `LoggerThread` will spawn a thread that listens to a mp.Queue
-    and logs messages from it with the provided logger,
-    so other processes can send logging messages to it
-    via the logger-like `SubLogger` interface.
-    the SubLogger even logs the pid of the caller.
+    The bare structlog loggers are thread-safe but not multiprocessing-safe. A
+    `LoggerThread` will spawn a thread that listens to a mp.Queue and logs
+    messages from it with the provided logger, so other processes can send
+    logging messages to it via the logger-like `SubLogger` interface. The
+    SubLogger even logs the pid of the caller.
 
-    this is good to use with a set of jobs that are part of a mp.Pool,
-    but isnt recommended for general use
-    because of overhead from threading and multiprocessing,
-    and because it might introduce lag to log messages.
+    This is good to use with a set of jobs that are part of a mp.Pool, but isnt
+    recommended for general use because of overhead from threading and
+    multiprocessing, and because it might introduce lag to log messages.
 
-    somewhat inspired by:
+    Somewhat inspired by:
     docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
     """
 
@@ -236,13 +236,11 @@ def pool_and_threadedlogger(logger, *poolargs):
     """
     Provide (to a context) a multiprocessing Pool and a proxy to the supplied logger.
 
-    Emulates the multiprocessing.Pool() context manager,
-    but also provides (via a LoggerThread) a SubLogger proxy to logger
-    that can be safely used by pool workers.
-    The SubLogger proxy interface supports these methods: debug, info, warning, error,
-    and critical.
-    Also "cleans up" the pool by waiting for workers to complete
-    as it exits the context.
+    Emulates the multiprocessing.Pool() context manager, but also provides (via
+    a LoggerThread) a SubLogger proxy to logger that can be safely used by pool
+    workers. The SubLogger proxy interface supports these methods: debug, info,
+    warning, error, and critical. Also "cleans up" the pool by waiting for
+    workers to complete as it exits the context.
     """
     with multiprocessing.Manager() as manager:
         logger_thread = LoggerThread(logger, manager.Queue())

diff --git a/_delphi_utils_python/pyproject.toml b/_delphi_utils_python/pyproject.toml
@@ -0,0 +1,57 @@
+[build-system]
+requires = ["setuptools", "setuptools-scm>=8.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "delphi-utils"
+version = "0.3.25"
+description = "Shared Utility Functions for Indicators"
+readme = "README.md"
+requires-python = "== 3.8.*"
+license = { text = "MIT License" }
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.8",
+    "License :: MIT",
+]
+dependencies = [
+    "boto3",
+    "covidcast",
+    "cvxpy",
+    "epiweeks",
+    "gitpython",
+    "importlib_resources>=1.3",
+    "numpy",
+    "pandas>=1.1.0",
+    "requests",
+    "slackclient",
+    "scs<3.2.6",                # TODO: remove this ; it is a cvxpy dependency, and the excluded version appears to break our jenkins build. see: https://github.com/cvxgrp/scs/issues/283
+    "structlog",
+    "xlrd",                     # needed by Pandas to read Excel files
+]
+
+[project.urls]
+Homepage = "https://github.com/cmu-delphi/covidcast-indicators"
+
+[project.optional-dependencies]
+dev = [
+    "darker[isort]~=2.1.1",
+    "pylint==2.8.3",
+    "pytest",
+    "pydocstyle",
+    "pytest-cov",
+    "mock",
+    "moto~=4.2.14",
+    "requests-mock",
+    "freezegun",
+]
+flash = ["scipy"]
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["delphi_utils"]
+namespaces = true
+
+[tool.setuptools.package-data]
+"delphi_utils.data" = ["20*/*.csv"]