CSV configuration variables

.devcontainer/.env.sample

@@ -1,2 +1,7 @@
 IMPORT_FILE_PATH=data/server.json
 INPUT_HASH_ALGO=sha256
+SUB_FORMAT_REGEX=test\\
+CSV_NEWLINE=
+CSV_DELIMITER=";"
+CSV_QUOTING=3
+CSV_QUOTECHAR=""

docs/getting-started.md

@@ -22,11 +22,27 @@ cd .devcontainer
 cp .env.sample .env
 ```
 
-The .env file specifies two values:
+The .env file specifies the following values:
 
-- the `IMPORT_FILE_PATH`: Must be either CSV or JSON.
+- `IMPORT_FILE_PATH`*: Must be either CSV or JSON.
 - `INPUT_HASH_ALGO`: Must be one of the types available in the [`hashlib` library's `algorithms_available` function](https://docs.python.org/3/library/hashlib.html#hashlib.algorithms_available).
 
+When using a CSV file, the following variables can be configured:
+
+- `CSV_DELIMITER`: specify a custom delimiter or use the default ","
+- `CSV_NEWLINE`: specify a newline or use the default of ""
+- `CSV_QUOTECHAR`: specify a quote character or use the default of none
+- `CSV_QUOTING`: default of 3 (no quotes)
+
+These are the possible values for the `CSV_QUOTING` variable:
+
+- `csv.QUOTE_MINIMAL`: 0 - To be used when the CSV file has quotes around entries which contain special characters such as delimiters, quotechar or any of the characters in lineterminator
+- `csv.QUOTE_ALL`: 1 - To be used when all the values in the CSV file are present inside quotation marks
+- `csv.QUOTE_NONNUMERIC`: 2 - To be used when the CSV file uses quotes around non-numeric entries
+- `csv.QUOTE_NONE`: 3 - To be used when the CSV file does not use quotes around entries
+
+Asterisk * indicates required
+
 ### Build image using Docker Compose
 
 ```bash
@@ -63,39 +79,39 @@ This repository comes with a [VS Code Remote Containers](https://code.visualstud
 Once you clone the repository locally, open it within VS Code, which will prompt you to re-open the repository within the Remote Container.
 
  1. Build and Open the Dev Container
- 2. Start the `eligibility-server` app with `F5`
- 3. Run any database scripts and test commands from within the container
+ 2. Start the `eligibility-server` Flask app and database with `F5`
+ 3. Now you can run tests from the container.
 
-## Set up database and run tests
+Starting the Dev Container will run `bin/start.sh`, which runs `setup.py` and starts the Flask app. The `setup.py` script creates a table, imports and saves users from a JSON or CSV file specified in the .env file from the `IMPORT_FILE_PATH` key. CSV files will require
 
-To run the API, you will have to set up the database first.
+## Run tests
 
-### Create and destroy database
+### Run unit tests
 
-The set up script creates a table, imports and saves users from a JSON or CSV file specified in the .env file from the `IMPORT_FILE_PATH` key. Each user has a name (string), key (string) and an array of eligibility types (string).
+Unit tests are implemented with [`pytest`](https://docs.pytest.org/en/6.2.x/) and can be found in the [`tests/`](https://github.com/cal-itp/eligibility-server/tree/main/tests) directory in the repository. `pytest` is installed and available to run directly in the devcontainer.
 
-To set up the database, run:
+The test suite runs against every pull request via a GitHub Action.
 
-```bash
-python setup.py
-```
+There are two different .env files to test against, to ensure the tests cover different `INPUT_HASH_ALGO` and `INPUT_FILE_PATH` types. To run tests on both files:
+
+1. From the main directory, run `coverage run -m pytest -m databasetest; coverage run -m pytest -m settingstest`
+2. To see the test coverage report, run `coverage report -m`
+
+### Destroy and recreate database
+
+In testing the database, you may need to teardown the database and restart a database from scratch.
 
 The teardown script removes all users and drops the database. To tear down the database, run:
 
 ```bash
 python teardown.py
 ```
 
-### Run unit tests
-
-Unit tests are implemented with [`pytest`](https://docs.pytest.org/en/6.2.x/) and can be found in the [`tests/`](https://github.com/cal-itp/eligibility-server/tree/main/tests) directory in the repository. `pytest` is installed and available to run directly in the devcontainer.
-
-The test suite runs against every pull request via a GitHub Action.
-
-There are two different .env files to test against, to ensure the tests cover different `INPUT_HASH_ALGO` and `INPUT_FILE_PATH` types. To run tests on both files:
+To set up the database with a new import file or other configuration variables, after making any new environment variable changes, run:
 
-1. From the main directory, run `coverage run -m pytest -m databasetest; coverage run -m pytest -m settingstest`
-2. To see the test coverage report, run `coverage report -m`
+```bash
+python setup.py
+```
 
 ## Run and develop the Documentation
 

eligibility_server/settings.py

@@ -28,3 +28,9 @@
 # Hash Configs from .env file
 
 INPUT_HASH_ALGO = os.environ.get("INPUT_HASH_ALGO", "")
+
+# CSV Configs from .env file
+CSV_DELIMITER = os.environ.get("CSV_DELIMITER", ",")
+CSV_NEWLINE = os.environ.get("CSV_NEWLINE", "")
+CSV_QUOTING = os.environ.get("CSV_QUOTING", 3)
+CSV_QUOTECHAR = os.environ.get("CSV_QUOTECHAR", "")

setup.py

@@ -6,15 +6,28 @@
 
 
 def import_users():
+    """
+    Imports user data to be added to database and saves user to database
+
+    Users can be imported from either a JSON file or CSV file, as configured
+    with settings from environment variables. CSV files take extra setting
+    configurations: CSV_DELIMITER, CSV_NEWLINE, CSV_QUOTING, CSV_QUOTECHAR
+    """
+
     print("Importing users from", settings.IMPORT_FILE_PATH)
     if settings.IMPORT_FILE_FORMAT == "json":
         with open(settings.IMPORT_FILE_PATH) as file:
             data = json.load(file)["users"]
             for user in data:
                 save_users(user, data[user][0], str(data[user][1]))
     elif settings.IMPORT_FILE_FORMAT == "csv":
-        with open(settings.IMPORT_FILE_PATH, newline="", encoding="utf-8") as file:
-            data = csv.reader(file, delimiter=";", quotechar="", quoting=csv.QUOTE_NONE)
+        with open(settings.IMPORT_FILE_PATH, newline=settings.CSV_NEWLINE, encoding="utf-8") as file:
+            data = csv.reader(
+                file,
+                delimiter=settings.CSV_DELIMITER,
+                quoting=int(settings.CSV_QUOTING),
+                quotechar=settings.CSV_QUOTECHAR,
+            )
             for user in data:
                 save_users(user[0], user[1], user[2])
     else:

tests/.env.settingstest

@@ -1,3 +1,7 @@
 IMPORT_FILE_PATH=data/server.csv
-INPUT_HASH_ALGO=sha512
+INPUT_HASH_ALGO=sha256
 SUB_FORMAT_REGEX=test\\
+CSV_NEWLINE=
+CSV_DELIMITER=";"
+CSV_QUOTING=3
+CSV_QUOTECHAR=

tests/test_settings.py

@@ -25,7 +25,7 @@ def test_hash_settings():
 
 @pytest.mark.settingstest
 def test_hash_settings_env():
-    assert settings.INPUT_HASH_ALGO == "sha512"
+    assert settings.INPUT_HASH_ALGO == "sha256"
     assert settings.IMPORT_FILE_FORMAT == "csv"
     assert settings.IMPORT_FILE_PATH == "data/server.csv"