add Conrad's test-fixture (built, run, smoke-tested) (bcgov#634)

legal-test-fixture/.dockerignore

@@ -0,0 +1,3 @@
+env
+.dockerignore
+Dockerfile

legal-test-fixture/Dockerfile

@@ -0,0 +1,20 @@
+# Base image
+FROM python:3.7
+
+# Update installation utilites and packages
+RUN apt-get update
+
+# Set working directory
+RUN mkdir /opt/server
+WORKDIR /opt/server
+
+# Add and install requirements
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Add rest of files
+COPY . .
+
+# Run server
+EXPOSE 5000
+CMD python manage.py run -h 0.0.0.0

legal-test-fixture/README.md

@@ -0,0 +1,94 @@
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+
+
+# Application Name
+
+Test Fixture API
+
+## Purpose
+In order to run test scenarios repeatedly, a known database state is required at the start of each test. This is surprisingly hard to set up in the current architecture. There are a limited number of businesses set up in the auth service and the pay service, so once they are affected by testing they are no longer usable for specific test scenarios. Many user activities are "one-way" in the sense that they can not be undone using the GUI for the application.
+
+All the current data loading is currently scripted by Python and requires an Oracle COLIN database instance as a source. This is not great for local development as nobody really runs Oracle locally (because it is a beast). QA team members are not necessarily equipped to run Python and would also need to manually reset the Oracle database before running a reload. The database loads are configured as all or nothing, which means single records can't be reset.
+
+Additionally, it is not at all simple for QA to inspect the current state of the data for a business, or to write a test that can set the precondition state of a business and know exactly the values it should be expecting. 
+
+This API serves the need for a simple tool that allows for bulk import/export of individual records or entire data sets for the PostgreSQL database. The specific use case that this API was designed to enable is to "snapshot" to file a known state for a business (maybe right after it was reset from COLIN) and then be able to "reset" that business back to that state whenever necessary. Because it is an API, a NightWatch test can be configured to execute the "reset" before (or after) each execution of a test. We can collect or create as many states as we like and check them into the repo in order to maintain a collection of data states for testing purposes.
+
+## Entities affected
+* Businesses
+* Business addresses
+* Directors
+* Director addresses
+* Filings
+
+## Limitations
+* Not intended to work with huge datasets
+* Does not maintain the state of the historical versioning of records (flask-continuum)
+* Does not in any way interact with the auth database or the payments database
+* Piggybacks off the current model. If there are bugs in the model, it will show up here. For example, the model currently validates for business identifiers that start with "CP" or "XCP", so a "BC" (benefit corporation) cannot be imported.
+
+
+## Technology Stack Used
+* Python, Flask, xlrd, xlwt
+* Postgres -  SQLAlchemy 
+
+## Files in this repository
+
+```
+legal-api/							- source dicrectory
+└── api/  
+    └── blueprints/ 
+            fixture.py				- contains the routes and main logic      
+    └── converter/           
+            ExcelConverter.py		- converts an incoming spreadsheet into database records
+            ExcelWriter.py			- converts a list of businesses from sqlalchemy into a spreadsheet
+            JsonWriter.py			- converts a list of businesses from sqlalchemy into a json document
+            utils.py				- shared code (formatting methods)     
+test/                      
+└── spreadsheets/  
+    └── businesses.xls				- sample import file     
+__init.py__							- initialization script
+config.py							- config script
+```
+
+## Deployment (Local Development)
+This service is intended to be built by the `make` command and run as a Docker container. 
+
+There is a cheat sheet for the manual steps that are still required to download the code from GitHub and build the solution: [https://docs.google.com/document/d/1tj4UgPoi698vS7F6HA-vxNXuveEODyUzImTBmXsARlo]() Hopefully these manual steps can be refactored and/or scripted away over time.
+
+Specifically, the command `make local-project` triggers a copy of models, exceptions, and schemas.py from the lear-api service folder to the source folder of the test fixture API. This way, the model is re-used and kept up to date automagically. ;-)
+
+## Deployment (Connect from local environment to OpenShift)
+If you have access to a database in OpenShift, you can connect your local instance of the test fixture API to that database. **BE CAREFUL**
+
+The way to do this is to edit the file docker-compose.yml locally to inject a different value for `DATABASE_URL` environment variable to point to the database you want to import to and export from. In the example below, the local environment has port forwarded local host 65432 to the openshift pod and port that is running PostgreSQL.
+
+Example: `- DATABASE_URL=postgres://user5SJ:[email protected]:65432/lear`
+
+In this usage scenario, there is no guarantee that the model you are using locally matches the model of the remote database, so you must confirm this yourself. YMMV
+
+## Deployment (Deploy to OpenShift)
+It's just a Docker container. By setting the single environment variable, it can be deployed in a Pod and configured to interact with a PostgreSQL database instance. Obviously this should never be done in production. *Deferred to Jenkins (or GitHub Actions) pipeline as per the preferences of the team.*
+
+## Usage
+The following examples assume that the make file has been used to deploy to a local Docker network and that the Test Fixture API has been mapped to port 5005. This is the default configuration. The domain and port would change if we were connecting to a running instance somewhere else (like OpenShift).
+
+See businesses.xls in this repository for an example of a file that can be used for import (or just get yourself a new file by doing an export). 
+
+### Export
+Export requires a GET request and returns either JSON or a file that is a spreadsheet in the same format used by the import function. This can be called in a browser, from a script, or from PostMan. (Hint: in PostMan you can click the little down arrow next to the big blue "Send" button and select the option "Send and Download"). Export does not affect the records in the database in any way.
+
+* `http://localhost:5005/api/fixture/export/CP0000393` - Exports the business record from the database in the default format (JSON). 
+* `http://localhost:5005/api/fixture/export/CP0000393/excel` - Exports the business record from the database in the excel format (downloads a file)
+* `http://localhost:5005/api/fixture/export/all_YES_IM_SURE` - Exports the all the business records from the database in the default format (JSON). **Will probably blow up for a large data set.**
+* `http://localhost:5005/api/fixture/export/all_YES_IM_SURE/excel` - Exports the all the business records from the database in the excel format (downloads a file). **Will probably blow up for a large data set.**
+
+
+### Import
+Import requires a POST request with a spreadsheet attached with the key "file". This is easily accomplished using PostMan. There are a few options that can be used to change the behaviour.
+
+* `http://localhost:5005/api/fixture/import` - Imports all the business records from the spreadsheet. For each business identifier, the API deletes the business from the database (and all child records) and rebuilds it from the data in the spreadsheet.
+* `http://localhost:5005/api/fixture/import/CP0000393` - Imports only the business record from the spreadsheet with the business ID "CP0000393". The API deletes this single business from the database (and all child records) and rebuilds it from the data in the spreadsheet. This is probably the most useful command for NightWatch scripts as they can save a state and reload it before each run of the test to ensure a known state.
+* `http://localhost:5005/api/fixture/import?rebuild=true` - **First DROPS all the records from the database and rebuilds the database according to the SqlAlchemy model.** Imports all the business records from the spreadsheet. 
+* `http://localhost:5005/api/fixture/import?rebuild=true` - **First DROPS all the records from the database and rebuilds the database according to the SqlAlchemy model.** Imports a single business record from the spreadsheet. There will only be one business in the database after this operation.

legal-test-fixture/legal_test_api/__init__.py

@@ -0,0 +1,33 @@
+import os
+from flask import Flask, jsonify
+from flask_sqlalchemy import SQLAlchemy
+from flask_cors import CORS
+from legal_api.models import db
+
+# Instantiate the database
+
+
+def create_app(script_info=None):
+    # Instantiate the app
+    app = Flask(__name__)
+
+    # Enable CORS
+    CORS(app)
+
+    # Get config
+    app.config.from_object('legal_test_api.config.Config')
+
+    # Set up extensions
+    db.init_app(app)
+
+    # Register blueprints
+    from legal_test_api.api.blueprints.fixture import fixture_blueprint
+    app.register_blueprint(fixture_blueprint)
+    # ADD OTHER BLUEPRINTS AS NEW RESOURCES ARE NEEDED
+
+    # Shell context for flask cli
+    @app.shell_context_processor
+    def ctx():
+        return {'app': app, 'db': db}
+
+    return app

legal-test-fixture/legal_test_api/api/blueprints/fixture.py

@@ -0,0 +1,98 @@
+from flask import Blueprint, jsonify, request, send_file
+from sqlalchemy import exc
+from legal_api import db
+from legal_api.models.business import Business, Director, Address
+from legal_api.models.office import OfficeType
+from legal_test_api.api.converter.ExcelConverter import ExcelConverter
+from legal_test_api.api.converter.ExcelWriter import ExcelWriter
+from legal_test_api.api.converter.JsonConverter import JsonConverter
+from datetime import datetime
+from http import HTTPStatus
+import logging
+import xlrd
+import io
+
+fixture_blueprint = Blueprint('fixture', __name__)
+
+
+@fixture_blueprint.route('/api/fixture/import/', methods=['POST'], strict_slashes=False, defaults={'business_identifier': ''})
+@fixture_blueprint.route('/api/fixture/import/<business_identifier>', methods=['POST'], strict_slashes=False)
+def post(business_identifier):
+    args = request.args
+    input_business_identifier = business_identifier
+
+    # If we are rebuilding, drop the db and recreate from sqlalchemy
+    rebuild = False
+    rebuild_arg_name = 'rebuild'
+    if rebuild_arg_name in args:
+        rebuild_arg_value = args[rebuild_arg_name]
+        rebuild_true_value = 'true'
+        rebuild = rebuild_true_value == rebuild_arg_value
+        if rebuild:
+            logging.warning('Rebuilding database')
+            db.drop_all()
+            db.create_all()
+            # Create lookup values
+            registered_office_type = OfficeType(
+                identifier=OfficeType.REGISTERED,
+                description=OfficeType.REGISTERED
+            )
+            db.session.add(registered_office_type)
+            records_office_type = OfficeType(
+                identifier=OfficeType.RECORDS,
+                description=OfficeType.RECORDS
+            )
+            db.session.add(records_office_type)
+            db.session.commit()
+
+    # return "{businesses:[]}"
+
+    # Open the workbook from the uploaded file
+    file_form_attribute_name = 'file'
+    a_file = request.files[file_form_attribute_name]
+    excel_converter = ExcelConverter()
+    business_list = excel_converter.create_businesses_from_file(
+        a_file, input_business_identifier, rebuild)
+
+    json_converter = JsonConverter()
+    return json_converter.convert_to_json(business_list)
+
+
+@fixture_blueprint.route('/api/fixture/export/<business_identifier>', methods=['GET'], strict_slashes=False, defaults={'format': 'JSON'})
+@fixture_blueprint.route('/api/fixture/export/<business_identifier>/<format>', methods=['GET'], strict_slashes=False)
+def get_all(business_identifier, format):
+
+    business_list = []
+
+    export_all_businesses_indicator = 'all_YES_IM_SURE'
+    if(business_identifier == export_all_businesses_indicator):
+        business_list = Business.query.all()
+    else:
+        business = Business.find_by_identifier(business_identifier)
+        if not business:
+            return jsonify({'message': f'{business_identifier} not found'}), HTTPStatus.NOT_FOUND
+        business_list.append(business)
+
+    excel_format_name = 'excel'
+    if(format == excel_format_name):
+        buf = __create_excel_file(business_list)
+        excel_mimetype = 'application/vnd.ms-excel'
+        return send_file(
+            buf,
+            as_attachment=True,
+            attachment_filename='%s.xls' % business_identifier,
+            mimetype=excel_mimetype
+        )
+
+    else:
+        json_converter = JsonConverter()
+        return json_converter.convert_to_json(business_list)
+
+
+def __create_excel_file(business_list):
+    excel_writer = ExcelWriter()
+    excel_object = excel_writer.convert_to_excel(business_list)
+    buf = io.BytesIO()
+    excel_object.save(buf)
+    buf.seek(0)
+    return buf