Make direnv optional (#48)

* Amend `.gitignore`s for `.env`, and `.envrc`

* Refactor environment variables in `.env`, `.secrets`, and `.envrc`

To make `direnv` optional, move all variables from `.envrc` to
`.env` in a `variable=value` format. Refactor `.secrets` to the
same format.

Use the `dotenv` and `dotenv_if_exists` `direnv` functions to
load both `.env` and `.secrets`, as `direnv` will be optional,
but still useful for folks who have it installed!

* Update tests due to changes to `.envrc` and `.env`

Also added `python-dotenv` as a requirement to enable this
testing, and the use of `.env` and `.secrets` going forward.

* Update README, structure/README and loading_environment_variabels for python-dotenv change

* Update loading_environment_variables.md for python-dotenv

* Fixing broken link in loading environment variable

Fixing broken link in loading environment variables

* Fix typo in documentation

* Update requirments

Add coverage[toml] to requirements

* fix internal link in contributing

* Address comments after peer review

* Remove coverage[toml] from requirments

* Add window make equivalent (#58)

Add make.bat file and update flake8 pre-commit hook

Co-authored-by: ESKYoung <[email protected]>

.gitignore

@@ -120,7 +120,6 @@ celerybeat.pid
 *.sage.py
 
 # Environments
-.env
 .venv
 env/
 venv/

.pre-commit-config.yaml

@@ -25,7 +25,7 @@ repos:
         name: black - consistent Python code formatting (auto-fixes)
         language_version: python # Should be a command that runs python3.6+
   - repo: https://github.com/PyCQA/flake8
-    rev: 3.9.2
+    rev: 5.0.4
     hooks:
       - id: flake8
         name: flake8 - Python linting

README.md

@@ -16,9 +16,6 @@ defined in the Aqua Book][aqua-book].
 
 ## Getting started
 
-[```Windows users should read this GitHub issue around govcookiecutter functionality```][issue20].
-
-
 [First, make sure your system meets the
 requirements](#requirements-to-create-a-cookiecutter-template). Next, open your
 terminal, navigate to the directory where you want your new repository to exist. Then

requirements.txt

@@ -9,5 +9,6 @@ pre-commit
 pytest
 pytest-mock
 pytest-xdist
+python-dotenv
 Sphinx
 toml

tests/test_envrc.py → tests/test_env.py

@@ -1,9 +1,10 @@
+from dotenv import dotenv_values
 from pathlib import Path
 from typing import Dict, List
 
-# Define a path to the `govcookiecutter` template directory, and its `.envrc` file
+# Define a path to the `govcookiecutter` template directory, and its `.env` file
 DIR_TEMPLATE = Path("{{ cookiecutter.repo_name }}")
-PATH_TEMPLATE_ENVRC = DIR_TEMPLATE.joinpath(".envrc")
+PATH_TEMPLATE_ENV = DIR_TEMPLATE.joinpath(".env")
 
 # Define a list of directory names to recursively ignore, as well as a list of
 # directory names to ignore at the root-level of the `govcookiecutter` template
@@ -13,45 +14,34 @@
 EXCLUDE_SUB_DIR_IN_PARENTS_NAMES = [*EXCLUDE_ROOT_DIR_NAMES, "docs"]
 
 
-def get_actual_envrc_variables(path_envrc: Path) -> Dict[str, Path]:
-    """Get the export variables and values for directories in the `.envrc` file of the
-    `govcookiecutter` template.
+def get_actual_env_variables(path_env: Path) -> Dict[str, Path]:
+    """Get the environment variables and values for directories in the `.env` file of
+     the `govcookiecutter` template.
 
     Args:
-        path_envrc: A file path to the `.envrc` file of the `govcookiecutter` template.
+        path_env: A file path to the `.env` file of the `govcookiecutter` template.
 
     Returns:
-        A dictionary where the keys are the names of the export directory variables,
-        and the values are the export directory values.
+        A dictionary where the keys are the names of the directory variables, and the
+        values are the directory paths.
 
     """
+    directory_variables = {}
+    for variable, value in dotenv_values(path_env).items():
+        if variable.startswith("DIR_"):
+            path_value = path_env.parent.joinpath(value).resolve()
+            directory_variables[variable] = path_value.relative_to(Path.cwd())
+    return directory_variables
+
 
-    # Instantiate a saving dictionary variable
-    envrc_actual_dir_variable = {}
-
-    # Open the `path_envrc` file, and get all export variables starting with `DIR`, and
-    # the values assigned to them, and return the output
-    with path_envrc.open() as f:
-        for line in f.readlines():
-            if line.startswith("export DIR"):
-                k, v = (
-                    line.lstrip("export ")
-                    .strip()
-                    .replace("$(pwd)", path_envrc.parent.name)
-                    .split("=")
-                )
-                envrc_actual_dir_variable[k] = Path(v)
-    return envrc_actual_dir_variable
-
-
-def define_expected_envrc_variables(
+def define_expected_env_variables(
     folder: Path,
     exclude_folders: List[str],
     exclude_root_folders: List[str],
     exclude_sub_folders_in_parent_folders: List[str],
 ) -> Dict[str, Path]:
-    """Get the expected export directory variables and values in the `.envrc` file of
-    the `govcookiecutter` template.
+    """Get the expected directory variables and values in the `.env` file of the
+    `govcookiecutter` template.
 
     Args:
         folder: A folder path to the `govcookiecutter` template folder.
@@ -61,16 +51,16 @@ def define_expected_envrc_variables(
             of `folder` where their sub-folders should be ignored.
 
     Returns:
-        A dictionary where the keys are expected export directory variables, and the
-        values are the expected export directory values of the `.envrc` file in the
-        `govcookiecutter` template directory `folder`.
+        A dictionary where the keys are expected directory variables, and the values
+        are the expected directory paths of the `.env` file in the `govcookiecutter`
+        template directory `folder`.
 
     """
 
     # Get the names of all root-level directories in `folder`, where each name is in
     # upper case in the format "DIR_<<<DIRECTORY_NAME>>>". Ignore any directories
     # with a name in `exclude_root_folders`
-    envrc_expected_dir_variable = {
+    env_expected_dir_variable = {
         f"DIR_{d.name.upper()}": d
         for d in folder.glob("*")
         if d.is_dir() and d.name not in exclude_root_folders
@@ -80,35 +70,34 @@ def define_expected_envrc_variables(
     # name is not in `exclude_folders`, and the parent directory name is not in
     # `exclude_sub_folders_in_parent_folders`. Each name is in upper case in the
     # format "DIR_<<<PARENT_DIRECTORY_NAME>>>_<<<DIRECTORY_NAME>>>". Return
-    # `envrc_expected_dir_variable`
+    # `env_expected_dir_variable`
     for d in folder.glob("*/*"):
         if (
             d.is_dir()
             and d.name not in exclude_folders
             and d.parent.name not in exclude_sub_folders_in_parent_folders
         ):
-            envrc_expected_dir_variable[
+            env_expected_dir_variable[
                 f"DIR_{d.parent.name.upper()}_{d.name.upper()}"
             ] = d
-    return envrc_expected_dir_variable
+    return env_expected_dir_variable
 
 
-class TestEnvrcExportDirectories:
+class TestEnvExportDirectories:
     def test_values_are_directories(self) -> None:
-        """Test that the export values are real dictionaries."""
-        for n, d in get_actual_envrc_variables(PATH_TEMPLATE_ENVRC).items():
+        """Test that the values are real directories."""
+        for n, d in get_actual_env_variables(PATH_TEMPLATE_ENV).items():
             assert d.is_dir(), f"`{n}` variable directory does not exist: {d}"
 
     def test_variables_as_expected(self) -> None:
-        """Test that the export directory variable and value names in `.envrc` are as
-        expected."""
+        """Test directory variable and value names in `.env` are as expected."""
 
-        # Get the expected `.envrc` directory variables, and assert the actual
+        # Get the expected `.env` directory variables, and assert the actual
         # variables are as expected
-        test_expected = define_expected_envrc_variables(
+        test_expected = define_expected_env_variables(
             DIR_TEMPLATE,
             EXCLUDE_DIR_NAMES,
             EXCLUDE_ROOT_DIR_NAMES,
             EXCLUDE_SUB_DIR_IN_PARENTS_NAMES,
         )
-        assert get_actual_envrc_variables(PATH_TEMPLATE_ENVRC) == test_expected
+        assert get_actual_env_variables(PATH_TEMPLATE_ENV) == test_expected

{{ cookiecutter.repo_name }}/.env

@@ -0,0 +1,50 @@
+# Environment variables go here, and can be read in by Python using the `python-dotenv`
+# package, and `os.getenv`:
+#
+#   ------------------------------------------------------------------------------------
+#   from dotenv import load_dotenv
+#   import os
+#
+#   # Load the environment variables from the `.env` file, overriding any system
+#   # environment variables
+#   load_dotenv(override=True)
+#
+#   # Load secrets from the `.secrets` file, overriding any system environment variables
+#   load_dotenv(".secrets", override=True)
+#
+#   # Example variable
+#   EXAMPLE_VARIABLE = os.getenv("EXAMPLE_VARIABLE")
+#   ------------------------------------------------------------------------------------
+#
+# For folder/file path environment variables, use relative paths.
+#
+# DO NOT STORE SECRETS HERE - this file is version-controlled! You should store secrets
+# in the untracked `.secrets` file.
+
+
+# Add environment variables for the `data` directories
+DIR_DATA=./data
+DIR_DATA_EXTERNAL=./data/external
+DIR_DATA_RAW=./data/raw
+DIR_DATA_INTERIM=./data/interim
+DIR_DATA_PROCESSED=./data/processed
+
+# Add environment variables for the `docs` directory
+DIR_DOCS=./docs
+
+# Add environment variables for the `notebooks` directory
+DIR_NOTEBOOKS=./notebooks
+
+# Add environment variables for the `outputs` directory
+DIR_OUTPUTS=./outputs
+
+# Add environment variables for the `src` directories
+DIR_SRC=./src
+DIR_SRC_MAKE_DATA=./src/make_data
+DIR_SRC_MAKE_FEATURES=./src/make_features
+DIR_SRC_MAKE_MODELS=./src/make_models
+DIR_SRC_MAKE_VISUALISATIONS=./src/make_visualisations
+DIR_SRC_UTILS=./src/utils
+
+# Add environment variables for the `tests` directory
+DIR_TESTS=./tests

{{ cookiecutter.repo_name }}/.envrc

@@ -1,52 +1,23 @@
-# Environment variables go here, and can be read in by Python using `os.getenv`:
+# Orchestration file to load environment variables from the `.env` and `.secrets` files.
 #
-#   --------------------------------------------------------
+# Only used by systems with `direnv` (https://direnv.net/) installed. Environment
+# variables can be read in by Python using `os.getenv` _without_ using `python-dotenv`:
+#
+#   ------------------------------------------------------------------------------------
 #   import os
 #
 #   # Example variable
 #   EXAMPLE_VARIABLE = os.getenv("EXAMPLE_VARIABLE")
-#   --------------------------------------------------------
-#
-# To ensure the `sed` command below works correctly, make sure all file paths in environment variables are absolute
-# (recommended), or are relative paths using other environment variables (works for Python users only). Environment
-# variable names are expected to contain letters, numbers or underscores only.
+#   ------------------------------------------------------------------------------------
 #
-# DO NOT STORE SECRETS HERE - this file is version-controlled! You should store secrets in a `.secrets` file, which is
-# not version-controlled - this can then be sourced here, using `source_env ".secrets"`.
-
-# Extract the variables to `.env` if required. Note `.env` is NOT version-controlled, so `.secrets` will not be
-# committed
-sed -n 's/^export \(.*\)$/\1/p' .envrc .secrets | sed -e 's?$(pwd)?'"$(pwd)"'?g' | sed -e 's?$\([a-zA-Z0-9_]\{1,\}\)?${\1}?g' > .env
+# DO NOT STORE SECRETS HERE - this file is version-controlled! You should store secrets
+# in the untracked `.secrets` file. This is loaded here using the `dotenv_if_exists`
+# command.
 
-# Add the working directory to `PYTHONPATH`; allows Jupyter notebooks in the `notebooks` folder to import `src`
+# Add the working directory to `PYTHONPATH`; allows Jupyter notebooks in the
+# `notebooks` folder to import `src`
 export PYTHONPATH="$PYTHONPATH:$(pwd)"
 
-# Import secrets from an untracked file `.secrets`
-source_env ".secrets"
-
-# Add environment variables for the `data` directories
-export DIR_DATA=$(pwd)/data
-export DIR_DATA_EXTERNAL=$(pwd)/data/external
-export DIR_DATA_RAW=$(pwd)/data/raw
-export DIR_DATA_INTERIM=$(pwd)/data/interim
-export DIR_DATA_PROCESSED=$(pwd)/data/processed
-
-# Add environment variables for the `docs` directory
-export DIR_DOCS=$(pwd)/docs
-
-# Add environment variables for the `notebooks` directory
-export DIR_NOTEBOOKS=$(pwd)/notebooks
-
-# Add environment variables for the `outputs` directory
-export DIR_OUTPUTS=$(pwd)/outputs
-
-# Add environment variables for the `src` directories
-export DIR_SRC=$(pwd)/src
-export DIR_SRC_MAKE_DATA=$(pwd)/src/make_data
-export DIR_SRC_MAKE_FEATURES=$(pwd)/src/make_features
-export DIR_SRC_MAKE_MODELS=$(pwd)/src/make_models
-export DIR_SRC_MAKE_VISUALISATIONS=$(pwd)/src/make_visualisations
-export DIR_SRC_UTILS=$(pwd)/src/utils
-
-# Add environment variables for the `tests` directory
-export DIR_TESTS=$(pwd)/tests
+# Load the `.env` file, and `.secrets` (if it exists)
+dotenv .env
+dotenv_if_exists .secrets

{{ cookiecutter.repo_name }}/.gitignore

@@ -120,7 +120,6 @@ celerybeat.pid
 *.sage.py
 
 # Environments
-.env
 .venv
 env/
 venv/

{{ cookiecutter.repo_name }}/.pre-commit-config.yaml

@@ -33,8 +33,8 @@ repos:
       - id: black
         name: black - consistent Python code formatting (auto-fixes)
         language_version: python # Should be a command that runs python3.6+
-  - repo: https://gitlab.com/pycqa/flake8
-    rev: 3.9.2
+  - repo: https://github.com/PyCQA/flake8
+    rev: 5.0.4
     hooks:
       - id: flake8
         name: flake8 - Python linting

{{ cookiecutter.repo_name }}/.secrets

@@ -1,15 +1,20 @@
 # Secrets and credentials should be stored here as environmental variables. For example:
 #
 #   # Google Cloud authentication credentials
-#   export GOOGLE_APPLICATION_CREDENTIALS="path/to/credentials.json"
+#   GOOGLE_APPLICATION_CREDENTIALS=path/to/credentials.json
 #
-# These environment variables can then be read in by Python using `os.getenv`:
+# These environment variables can then be read in by Python using the `python-dotenv`
+# package, and `os.getenv`:
 #
-#   --------------------------------------------------------
+#   ------------------------------------------------------------------------------------
+#   from dotenv import load_dotenv
 #   import os
 #
+#   # Load secrets from the `.secrets` file, overriding any system environment variables
+#   load_dotenv(".secrets", override=True)
+#
 #   # Google Cloud authentication credentials
 #   GOOGLE_APPLICATION_CREDENTIALS = os.getenv("GOOGLE_APPLICATION_CREDENTIALS")
-#   --------------------------------------------------------
+#   ------------------------------------------------------------------------------------
 #
 # This file is NOT version-controlled!

{{ cookiecutter.repo_name }}/README.md

@@ -24,7 +24,7 @@ To be added.
 {% endif -%}
 - a `.secrets` file with the [required secrets and
   credentials](#required-secrets-and-credentials)
-- [load environment variables][docs-loading-environment-variables] from `.envrc`
+- [load environment variables][docs-loading-environment-variables] from `.env`
 
 To install the Python requirements, open your terminal and enter:
 
@@ -44,7 +44,7 @@ secrets/credentials should have the following environment variable name(s):
 | Credential 1      | `CREDENTIAL_VARIABLE_1`   | Plain English description of Credential 1. |
 
 Once you've added, [load these environment variables using
-`.envrc`][docs-loading-environment-variables].
+`.env`][docs-loading-environment-variables].
 
 ## Licence