diff --git a/legal-test-fixture/.dockerignore b/legal-test-fixture/.dockerignore new file mode 100755 index 0000000000..60512ac536 --- /dev/null +++ b/legal-test-fixture/.dockerignore @@ -0,0 +1,3 @@ +env +.dockerignore +Dockerfile \ No newline at end of file diff --git a/legal-test-fixture/Dockerfile b/legal-test-fixture/Dockerfile new file mode 100755 index 0000000000..5024cbb525 --- /dev/null +++ b/legal-test-fixture/Dockerfile @@ -0,0 +1,20 @@ +# Base image +FROM python:3.7 + +# Update installation utilites and packages +RUN apt-get update + +# Set working directory +RUN mkdir /opt/server +WORKDIR /opt/server + +# Add and install requirements +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +# Add rest of files +COPY . . + +# Run server +EXPOSE 5000 +CMD python manage.py run -h 0.0.0.0 \ No newline at end of file diff --git a/legal-test-fixture/README.md b/legal-test-fixture/README.md new file mode 100644 index 0000000000..2f4ac4c8e1 --- /dev/null +++ b/legal-test-fixture/README.md @@ -0,0 +1,94 @@ + +[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) + + +# Application Name + +Test Fixture API + +## Purpose +In order to run test scenarios repeatedly, a known database state is required at the start of each test. This is surprisingly hard to set up in the current architecture. There are a limited number of businesses set up in the auth service and the pay service, so once they are affected by testing they are no longer usable for specific test scenarios. Many user activities are "one-way" in the sense that they can not be undone using the GUI for the application. + +All the current data loading is currently scripted by Python and requires an Oracle COLIN database instance as a source. This is not great for local development as nobody really runs Oracle locally (because it is a beast). QA team members are not necessarily equipped to run Python and would also need to manually reset the Oracle database before running a reload. The database loads are configured as all or nothing, which means single records can't be reset. + +Additionally, it is not at all simple for QA to inspect the current state of the data for a business, or to write a test that can set the precondition state of a business and know exactly the values it should be expecting. + +This API serves the need for a simple tool that allows for bulk import/export of individual records or entire data sets for the PostgreSQL database. The specific use case that this API was designed to enable is to "snapshot" to file a known state for a business (maybe right after it was reset from COLIN) and then be able to "reset" that business back to that state whenever necessary. Because it is an API, a NightWatch test can be configured to execute the "reset" before (or after) each execution of a test. We can collect or create as many states as we like and check them into the repo in order to maintain a collection of data states for testing purposes. + +## Entities affected +* Businesses +* Business addresses +* Directors +* Director addresses +* Filings + +## Limitations +* Not intended to work with huge datasets +* Does not maintain the state of the historical versioning of records (flask-continuum) +* Does not in any way interact with the auth database or the payments database +* Piggybacks off the current model. If there are bugs in the model, it will show up here. For example, the model currently validates for business identifiers that start with "CP" or "XCP", so a "BC" (benefit corporation) cannot be imported. + + +## Technology Stack Used +* Python, Flask, xlrd, xlwt +* Postgres - SQLAlchemy + +## Files in this repository + +``` +legal-api/ - source dicrectory +└── api/ + └── blueprints/ + fixture.py - contains the routes and main logic + └── converter/ + ExcelConverter.py - converts an incoming spreadsheet into database records + ExcelWriter.py - converts a list of businesses from sqlalchemy into a spreadsheet + JsonWriter.py - converts a list of businesses from sqlalchemy into a json document + utils.py - shared code (formatting methods) +test/ +└── spreadsheets/ + └── businesses.xls - sample import file +__init.py__ - initialization script +config.py - config script +``` + +## Deployment (Local Development) +This service is intended to be built by the `make` command and run as a Docker container. + +There is a cheat sheet for the manual steps that are still required to download the code from GitHub and build the solution: [https://docs.google.com/document/d/1tj4UgPoi698vS7F6HA-vxNXuveEODyUzImTBmXsARlo]() Hopefully these manual steps can be refactored and/or scripted away over time. + +Specifically, the command `make local-project` triggers a copy of models, exceptions, and schemas.py from the lear-api service folder to the source folder of the test fixture API. This way, the model is re-used and kept up to date automagically. ;-) + +## Deployment (Connect from local environment to OpenShift) +If you have access to a database in OpenShift, you can connect your local instance of the test fixture API to that database. **BE CAREFUL** + +The way to do this is to edit the file docker-compose.yml locally to inject a different value for `DATABASE_URL` environment variable to point to the database you want to import to and export from. In the example below, the local environment has port forwarded local host 65432 to the openshift pod and port that is running PostgreSQL. + +Example: `- DATABASE_URL=postgres://user5SJ:password@host.docker.internal:65432/lear` + +In this usage scenario, there is no guarantee that the model you are using locally matches the model of the remote database, so you must confirm this yourself. YMMV + +## Deployment (Deploy to OpenShift) +It's just a Docker container. By setting the single environment variable, it can be deployed in a Pod and configured to interact with a PostgreSQL database instance. Obviously this should never be done in production. *Deferred to Jenkins (or GitHub Actions) pipeline as per the preferences of the team.* + +## Usage +The following examples assume that the make file has been used to deploy to a local Docker network and that the Test Fixture API has been mapped to port 5005. This is the default configuration. The domain and port would change if we were connecting to a running instance somewhere else (like OpenShift). + +See businesses.xls in this repository for an example of a file that can be used for import (or just get yourself a new file by doing an export). + +### Export +Export requires a GET request and returns either JSON or a file that is a spreadsheet in the same format used by the import function. This can be called in a browser, from a script, or from PostMan. (Hint: in PostMan you can click the little down arrow next to the big blue "Send" button and select the option "Send and Download"). Export does not affect the records in the database in any way. + +* `http://localhost:5005/api/fixture/export/CP0000393` - Exports the business record from the database in the default format (JSON). +* `http://localhost:5005/api/fixture/export/CP0000393/excel` - Exports the business record from the database in the excel format (downloads a file) +* `http://localhost:5005/api/fixture/export/all_YES_IM_SURE` - Exports the all the business records from the database in the default format (JSON). **Will probably blow up for a large data set.** +* `http://localhost:5005/api/fixture/export/all_YES_IM_SURE/excel` - Exports the all the business records from the database in the excel format (downloads a file). **Will probably blow up for a large data set.** + + +### Import +Import requires a POST request with a spreadsheet attached with the key "file". This is easily accomplished using PostMan. There are a few options that can be used to change the behaviour. + +* `http://localhost:5005/api/fixture/import` - Imports all the business records from the spreadsheet. For each business identifier, the API deletes the business from the database (and all child records) and rebuilds it from the data in the spreadsheet. +* `http://localhost:5005/api/fixture/import/CP0000393` - Imports only the business record from the spreadsheet with the business ID "CP0000393". The API deletes this single business from the database (and all child records) and rebuilds it from the data in the spreadsheet. This is probably the most useful command for NightWatch scripts as they can save a state and reload it before each run of the test to ensure a known state. +* `http://localhost:5005/api/fixture/import?rebuild=true` - **First DROPS all the records from the database and rebuilds the database according to the SqlAlchemy model.** Imports all the business records from the spreadsheet. +* `http://localhost:5005/api/fixture/import?rebuild=true` - **First DROPS all the records from the database and rebuilds the database according to the SqlAlchemy model.** Imports a single business record from the spreadsheet. There will only be one business in the database after this operation. \ No newline at end of file diff --git a/legal-test-fixture/legal_test_api/__init__.py b/legal-test-fixture/legal_test_api/__init__.py new file mode 100755 index 0000000000..68d96eb44c --- /dev/null +++ b/legal-test-fixture/legal_test_api/__init__.py @@ -0,0 +1,33 @@ +import os +from flask import Flask, jsonify +from flask_sqlalchemy import SQLAlchemy +from flask_cors import CORS +from legal_api.models import db + +# Instantiate the database + + +def create_app(script_info=None): + # Instantiate the app + app = Flask(__name__) + + # Enable CORS + CORS(app) + + # Get config + app.config.from_object('legal_test_api.config.Config') + + # Set up extensions + db.init_app(app) + + # Register blueprints + from legal_test_api.api.blueprints.fixture import fixture_blueprint + app.register_blueprint(fixture_blueprint) + # ADD OTHER BLUEPRINTS AS NEW RESOURCES ARE NEEDED + + # Shell context for flask cli + @app.shell_context_processor + def ctx(): + return {'app': app, 'db': db} + + return app diff --git a/legal-test-fixture/legal_test_api/api/blueprints/fixture.py b/legal-test-fixture/legal_test_api/api/blueprints/fixture.py new file mode 100644 index 0000000000..e40ba10b79 --- /dev/null +++ b/legal-test-fixture/legal_test_api/api/blueprints/fixture.py @@ -0,0 +1,98 @@ +from flask import Blueprint, jsonify, request, send_file +from sqlalchemy import exc +from legal_api import db +from legal_api.models.business import Business, Director, Address +from legal_api.models.office import OfficeType +from legal_test_api.api.converter.ExcelConverter import ExcelConverter +from legal_test_api.api.converter.ExcelWriter import ExcelWriter +from legal_test_api.api.converter.JsonConverter import JsonConverter +from datetime import datetime +from http import HTTPStatus +import logging +import xlrd +import io + +fixture_blueprint = Blueprint('fixture', __name__) + + +@fixture_blueprint.route('/api/fixture/import/', methods=['POST'], strict_slashes=False, defaults={'business_identifier': ''}) +@fixture_blueprint.route('/api/fixture/import/', methods=['POST'], strict_slashes=False) +def post(business_identifier): + args = request.args + input_business_identifier = business_identifier + + # If we are rebuilding, drop the db and recreate from sqlalchemy + rebuild = False + rebuild_arg_name = 'rebuild' + if rebuild_arg_name in args: + rebuild_arg_value = args[rebuild_arg_name] + rebuild_true_value = 'true' + rebuild = rebuild_true_value == rebuild_arg_value + if rebuild: + logging.warning('Rebuilding database') + db.drop_all() + db.create_all() + # Create lookup values + registered_office_type = OfficeType( + identifier=OfficeType.REGISTERED, + description=OfficeType.REGISTERED + ) + db.session.add(registered_office_type) + records_office_type = OfficeType( + identifier=OfficeType.RECORDS, + description=OfficeType.RECORDS + ) + db.session.add(records_office_type) + db.session.commit() + + # return "{businesses:[]}" + + # Open the workbook from the uploaded file + file_form_attribute_name = 'file' + a_file = request.files[file_form_attribute_name] + excel_converter = ExcelConverter() + business_list = excel_converter.create_businesses_from_file( + a_file, input_business_identifier, rebuild) + + json_converter = JsonConverter() + return json_converter.convert_to_json(business_list) + + +@fixture_blueprint.route('/api/fixture/export/', methods=['GET'], strict_slashes=False, defaults={'format': 'JSON'}) +@fixture_blueprint.route('/api/fixture/export//', methods=['GET'], strict_slashes=False) +def get_all(business_identifier, format): + + business_list = [] + + export_all_businesses_indicator = 'all_YES_IM_SURE' + if(business_identifier == export_all_businesses_indicator): + business_list = Business.query.all() + else: + business = Business.find_by_identifier(business_identifier) + if not business: + return jsonify({'message': f'{business_identifier} not found'}), HTTPStatus.NOT_FOUND + business_list.append(business) + + excel_format_name = 'excel' + if(format == excel_format_name): + buf = __create_excel_file(business_list) + excel_mimetype = 'application/vnd.ms-excel' + return send_file( + buf, + as_attachment=True, + attachment_filename='%s.xls' % business_identifier, + mimetype=excel_mimetype + ) + + else: + json_converter = JsonConverter() + return json_converter.convert_to_json(business_list) + + +def __create_excel_file(business_list): + excel_writer = ExcelWriter() + excel_object = excel_writer.convert_to_excel(business_list) + buf = io.BytesIO() + excel_object.save(buf) + buf.seek(0) + return buf diff --git a/legal-test-fixture/legal_test_api/api/converter/ExcelConverter.py b/legal-test-fixture/legal_test_api/api/converter/ExcelConverter.py new file mode 100644 index 0000000000..addd49fb7d --- /dev/null +++ b/legal-test-fixture/legal_test_api/api/converter/ExcelConverter.py @@ -0,0 +1,256 @@ +from flask import json +from sqlalchemy import exc +from sqlalchemy_continuum import versioning_manager +from legal_api import db +from legal_api.models.business import Business, Director, Address, Filing +from legal_api.models.office import Office, OfficeType +from legal_test_api.api.converter.utils import format_date, format_non_date, format_boolean, SheetName +from datetime import datetime +from enum import Enum +import logging +import xlrd + + +class ExcelConverter(): + + def create_businesses_from_file(self, a_file, input_business_identifier, rebuild): + book = xlrd.open_workbook(file_contents=a_file.stream.read()) + + # Start to process the sheet with the businesses + business_sheet = book.sheet_by_name(SheetName.BUSINESS.value) + business_list = [] + + business_rows = list(business_sheet.get_rows()) + iterrows = iter(business_rows) + + # (skipping the header line) + next(iterrows) + + for row in iterrows: + row_business_identifier = self.__get_business_identifier(row) + + # If business_identifier is provided then only process this row if the BI matches + if input_business_identifier and not (input_business_identifier == row_business_identifier): + continue + + # If we are rebuilding, everything is already dropped. otherwise delete this single business + if not rebuild: + # If this business is already in the database, delete it + existing_business = Business.find_by_identifier( + row_business_identifier) + if existing_business: + + for f in existing_business.filings.all(): + f._payment_token = None + db.session.delete(f) + + for ma in existing_business.mailing_address.all(): + db.session.delete(ma) + for da in existing_business.delivery_address.all(): + db.session.delete(da) + for office in existing_business.offices.all(): + db.session.delete(office) + + db.session.delete(existing_business) + db.session.delete(existing_business) + + business = self.__create_business_from_row(row, book) + business_list.append(business) + return business_list + + def __get_business_identifier(self, row): + return self.__get_value_from_row(row, 0) + + def __create_business_from_row(self, row, book): + # Get the business properties and create the business + business = Business(identifier=self.__get_value_from_row(row, 0), + legal_name=self.__get_value_from_row(row, 1), + legal_type=self.__get_value_from_row(row, 2), + founding_date=self.__get_value_from_row(row, 3), + dissolution_date=self.__get_value_from_row(row, 4), + last_ar_date=self.__get_value_from_row(row, 5), + last_agm_date=self.__get_value_from_row(row, 6), + fiscal_year_end_date=self.__get_value_from_row( + row, 7), + tax_id=self.__get_value_from_row(row, 8), + last_ledger_id=self.__get_value_from_row(row, 9), + last_remote_ledger_id=self.__get_value_from_row( + row, 10), + last_ledger_timestamp=self.__get_value_from_row( + row, 11), + last_modified=self.__get_value_from_row(row, 12) + ) + business.save() + self.__add_directors(business, book) + self.__add_business_addresses(business, book) + self.__add_filings(business, book) + db.session.commit() + return business + + def __add_directors(self, business, book): + # Get the director properties and create the directors + director_sheet = book.sheet_by_name(SheetName.DIRECTOR.value) + iter_director_rows = iter(director_sheet.get_rows()) + # (skipping the header line) + next(iter_director_rows) + for director_row in iter_director_rows: + if director_row[0].value == business.identifier: + director = Director( + business_id=business.id, + first_name=self.__get_value_from_row(director_row, 1), + middle_initial=self.__get_value_from_row(director_row, 2), + last_name=self.__get_value_from_row(director_row, 3), + title=self.__get_value_from_row(director_row, 4), + appointment_date=self.__get_value_from_row( + director_row, 5), + cessation_date=self.__get_value_from_row(director_row, 6) + ) + self.__add_director_addresses( + business.identifier, director, book) + + business.directors.append(director) + db.session.add(director) + director.save() + + def __add_director_addresses(self, business_identifier, director, book): + # Find Mailing and Delivery Addresses + director_address_sheet = book.sheet_by_name( + SheetName.DIRECTOR_ADDRESS.value) + iter_director_address_rows = iter(director_address_sheet.get_rows()) + # (skipping the header line) + next(iter_director_address_rows) + for director_address_row in iter_director_address_rows: + da_business_identifier = self.__get_value_from_row( + director_address_row, 0) + da_first_name = self.__get_value_from_row(director_address_row, 1) + da_last_name = self.__get_value_from_row(director_address_row, 2) + if da_business_identifier == business_identifier and da_first_name == director.first_name and da_last_name == director.last_name: + address = Address( + address_type=self.__get_value_from_row( + director_address_row, 3), + street=self.__get_value_from_row(director_address_row, 4), + street_additional=self.__get_value_from_row( + director_address_row, 5), + city=self.__get_value_from_row(director_address_row, 6), + region=self.__get_value_from_row(director_address_row, 7), + country=self.__get_value_from_row(director_address_row, 8), + postal_code=self.__get_value_from_row( + director_address_row, 9), + delivery_instructions=self.__get_value_from_row( + director_address_row, 10) + ) + address.save() + if (address.address_type == Address.MAILING): + director.mailing_address = address + director.mailing_address_id = address.id + elif (address.address_type == Address.DELIVERY): + director.delivery_address = address + director.delivery_address_id = address.id + + # If the mailing anddress and the delivery address are both found, no need to continue + if director.mailing_address_id and director.delivery_address_id: + break + + def __add_business_addresses(self, business, book): + registered_office_type_id = OfficeType.REGISTERED + records_office_type_id = OfficeType.RECORDS + + # Get the business address properties and create the business addresses + business_address_sheet = book.sheet_by_name( + SheetName.BUSINESS_ADDRESS.value) + iter_business_address_rows = iter(business_address_sheet.get_rows()) + # (skipping the header line) + next(iter_business_address_rows) + registered_office = None + records_office = None + for business_address_row in iter_business_address_rows: + business_offices = business.offices.all() + + if business_address_row[0].value == business.identifier: + office_type=self.__get_value_from_row(business_address_row, 1) + office = None + + # Create the appropriate office if it does not exist + if (office_type == OfficeType.REGISTERED) and not registered_office: + # Create registered office + registered_office = Office( + business_id=business.id, + office_type=registered_office_type_id + ) + db.session.add(registered_office) + business_offices.append(registered_office) + elif (office_type == OfficeType.RECORDS) and not records_office: + # Create records office + records_office = Office( + business_id=business.id, + office_type=records_office_type_id + ) + db.session.add(records_office) + business_offices.append(records_office) + + # Set the office to use to save the address + if office_type == OfficeType.REGISTERED: + office = registered_office + elif office_type == OfficeType.RECORDS: + office = records_office + + address = Address( + address_type=self.__get_value_from_row(business_address_row, 2), + street=self.__get_value_from_row(business_address_row, 3), + street_additional=self.__get_value_from_row(business_address_row, 4), + city=self.__get_value_from_row(business_address_row, 5), + region=self.__get_value_from_row(business_address_row, 6), + country=self.__get_value_from_row(business_address_row, 7), + postal_code=self.__get_value_from_row(business_address_row, 8), + # delivery_instructions=self.__get_value_from_row(business_address_row, 9), + office_id=office.id, + business_id=business.id + ) + db.session.add(address) + office.addresses.append(address) + db.session.commit() + + def __add_filings(self, business, book): + # Get the filings properties and create the filings + filings_sheet = book.sheet_by_name(SheetName.FILING.value) + iter_filings_rows = iter(filings_sheet.get_rows()) + # (skipping the header line) + next(iter_filings_rows) + for filing_row in iter_filings_rows: + transaction_id = None + if filing_row[0].value == business.identifier: + + # If the filing is completed, it has to contain a transaction ID + status = self.__get_value_from_row(filing_row, 9) + if(Filing.Status.COMPLETED.value == status): + uow = versioning_manager.unit_of_work(db.session) + transaction = uow.create_transaction(db.session) + transaction_id = transaction.id + + filing = Filing( + _completion_date=self.__get_value_from_row(filing_row, 2), + _filing_date=self.__get_value_from_row(filing_row, 3), + _filing_type=self.__get_value_from_row(filing_row, 4), + effective_date=self.__get_value_from_row(filing_row, 5), + _payment_token=self.__get_value_from_row(filing_row, 6), + _payment_completion_date=self.__get_value_from_row( + filing_row, 7), + colin_event_id=self.__get_value_from_row(filing_row, 8), + _status=status, + paper_only=self.__get_value_from_row(filing_row, 10), + # transaction_id comes from continuuum + transaction_id=transaction_id + ) + filing.business_id = business.id + + # need to convert this first before storing + filing_value = self.__get_value_from_row(filing_row, 1) + if(filing_value): + filing._filing_json = json.loads(filing_value) + + business.filings.append(filing) + db.session.add(filing) + db.session.commit() + + def __get_value_from_row(self, row, index): + return row[index].value if row[index].value else None diff --git a/legal-test-fixture/legal_test_api/api/converter/ExcelWriter.py b/legal-test-fixture/legal_test_api/api/converter/ExcelWriter.py new file mode 100644 index 0000000000..5eff7b3c20 --- /dev/null +++ b/legal-test-fixture/legal_test_api/api/converter/ExcelWriter.py @@ -0,0 +1,278 @@ +from flask import jsonify +from datetime import datetime +from legal_api.models.business import Business, Director, Address, Filing +from legal_test_api.api.converter.utils import format_date, format_non_date, format_boolean, format_json, SheetName +import xlwt +import logging + + +class ExcelWriter(): + + __business_sheet = None + __business_address_sheet = None + __director_sheet = None + __director_address_sheet = None + __filing_sheet = None + + # row_num is offset by 1 because of the header row + __business_sheet_row_index = 1 + __business_address_sheet_row_index = 1 + __director_sheet_row_index = 1 + __director_address_sheet_row_index = 1 + __filing_sheet_row_index = 1 + + def convert_to_excel(self, business_list): + book = xlwt.Workbook(encoding='ascii') + + # Add the sheets + self.__business_sheet = book.add_sheet(SheetName.BUSINESS.value) + self.__business_address_sheet = book.add_sheet( + SheetName.BUSINESS_ADDRESS.value) + self.__director_sheet = book.add_sheet(SheetName.DIRECTOR.value) + self.__director_address_sheet = book.add_sheet( + SheetName.DIRECTOR_ADDRESS.value) + self.__filing_sheet = book.add_sheet(SheetName.FILING.value) + + # Write header lines + self.__write_header_lines() + + for business in business_list: + self.__write_business_to_excel(business) + + return book + + def __write_header_lines(self): + business_sheet_headings = [ + 'identifier', + 'legal_name', + 'legal_type', + 'founding_date', + 'dissolution_date', + 'last_ar_date', + 'last_agm_date', + 'fiscal_year_end_date', + 'tax_id', + 'last_ledger_id', + 'last_remote_ledger_id', + 'last_ledger_timestamp', + 'last_modified' + ] + for i, business_sheet_heading in enumerate(business_sheet_headings): + self.__business_sheet.write( + 0, i, format_non_date(business_sheet_heading)) + + business_address_sheet_headings = [ + 'business', + 'address_type', + 'street', + 'street_additional', + 'city', + 'region', + 'country', + 'postal_code', + 'delivery_instructions' + ] + for i, business_address_sheet_heading in enumerate(business_address_sheet_headings): + self.__business_address_sheet.write( + 0, i, format_non_date(business_address_sheet_heading)) + + director_sheet_headings = [ + 'business', + 'first_name', + 'middle_initial', + 'last_name', + 'title', + 'appointment_date', + 'cessation_date', + ] + for i, director_sheet_heading in enumerate(director_sheet_headings): + self.__director_sheet.write( + 0, i, format_non_date(director_sheet_heading)) + + director_address_sheet_headings = [ + 'business', + 'first_name', + 'last_name', + 'address_type', + 'street', + 'street_additional', + 'city', + 'region', + 'country', + 'postal_code', + 'delivery_instructions' + ] + for i, director_address_sheet_heading in enumerate(director_address_sheet_headings): + self.__director_address_sheet.write( + 0, i, format_non_date(director_address_sheet_heading)) + + filing_sheet_headings = [ + 'business', + 'filing_json', + 'completion_date', + 'filing_date', + 'filing_type', + 'effective_date', + 'payment_id', + 'payment_completion_date', + 'colin_event_id', + 'status', + 'paper_only', + ] + for i, filing_sheet_heading in enumerate(filing_sheet_headings): + self.__filing_sheet.write( + 0, i, format_non_date(filing_sheet_heading)) + + def __write_business_to_excel(self, business): + + self.__business_sheet.write( + self.__business_sheet_row_index, 0, format_non_date(business.identifier)) + self.__business_sheet.write( + self.__business_sheet_row_index, 1, format_non_date(business.legal_name)) + self.__business_sheet.write( + self.__business_sheet_row_index, 2, format_non_date(business.legal_type)) + self.__business_sheet.write( + self.__business_sheet_row_index, 3, format_date(business.founding_date)) + self.__business_sheet.write( + self.__business_sheet_row_index, 4, format_date(business.dissolution_date)) + self.__business_sheet.write( + self.__business_sheet_row_index, 5, format_date(business.last_ar_date)) + self.__business_sheet.write( + self.__business_sheet_row_index, 6, format_date(business.last_agm_date)) + self.__business_sheet.write( + self.__business_sheet_row_index, 7, format_date(business.fiscal_year_end_date)) + self.__business_sheet.write( + self.__business_sheet_row_index, 8, format_non_date(business.tax_id)) + self.__business_sheet.write( + self.__business_sheet_row_index, 9, format_non_date(business.last_ledger_id)) + self.__business_sheet.write(self.__business_sheet_row_index, 10, format_non_date( + business.last_remote_ledger_id)) + self.__business_sheet.write(self.__business_sheet_row_index, 11, format_date( + business.last_ledger_timestamp)) + self.__business_sheet.write( + self.__business_sheet_row_index, 12, format_date(business.last_modified)) + + self.__business_sheet_row_index += 1 + + directors = business.directors.all() + for director in directors: + self.__write_director_to_excel(business.identifier, director) + + offices = business.offices.all() + for office in offices: + office_addresses = office.addresses.all() + for office_address in office_addresses: + self.__write_business_address_to_excel(business.identifier, office.office_type, office_address) + + filings = business.filings.all() + for filing in filings: + self.__write_filing_to_excel(business.identifier, filing) + + def __write_director_to_excel(self, business_identifier, director): + self.__director_sheet.write( + self.__director_sheet_row_index, 0, format_non_date(business_identifier)) + self.__director_sheet.write( + self.__director_sheet_row_index, 1, format_non_date(director.first_name)) + self.__director_sheet.write( + self.__director_sheet_row_index, 2, format_non_date(director.middle_initial)) + self.__director_sheet.write( + self.__director_sheet_row_index, 3, format_non_date(director.last_name)) + self.__director_sheet.write( + self.__director_sheet_row_index, 4, format_non_date(director.title)) + self.__director_sheet.write( + self.__director_sheet_row_index, 5, format_date(director.appointment_date)) + self.__director_sheet.write( + self.__director_sheet_row_index, 6, format_date(director.cessation_date)) + + delivery_address = director.delivery_address + if (delivery_address): + self.__write_director_address_to_excel( + business_identifier, director, delivery_address, self.__director_sheet_row_index) + + mailing_address = director.mailing_address + if (mailing_address): + self.__write_director_address_to_excel( + business_identifier, director, mailing_address, self.__director_sheet_row_index) + + self.__director_sheet_row_index += 1 + + def __write_director_address_to_excel(self, business_identifier, director, director_address, director_row_reference): + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 0, format_non_date(business_identifier)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 1, format_non_date(director.first_name)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 2, format_non_date(director.middle_initial)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 3, format_non_date(director_address.address_type)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 4, format_non_date(director_address.street)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 5, format_non_date(director_address.street_additional)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 6, format_non_date(director_address.city)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 7, format_non_date(director_address.region)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 8, format_non_date(director_address.country)) + self.__director_address_sheet.write( + self.__director_address_sheet_row_index, 9, format_non_date(director_address.postal_code)) + self.__director_address_sheet.write(self.__director_address_sheet_row_index, 10, format_non_date( + director_address.delivery_instructions)) + # Need to add the row reference so that it can be linked with a specific director (matching by name was bad) + self.__director_address_sheet.write(self.__director_address_sheet_row_index, 11, format_non_date( + director_row_reference)) + + self.__director_address_sheet_row_index += 1 + + def __write_business_address_to_excel(self, business_identifier, office_type, business_address): + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 0, format_non_date(business_identifier)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 1, format_non_date(office_type)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 2, format_non_date(business_address.address_type)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 3, format_non_date(business_address.street)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 4, format_non_date(business_address.street_additional)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 5, format_non_date(business_address.city)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 6, format_non_date(business_address.region)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 7, format_non_date(business_address.country)) + self.__business_address_sheet.write( + self.__business_address_sheet_row_index, 8, format_non_date(business_address.postal_code)) + self.__business_address_sheet.write(self.__business_address_sheet_row_index, 9, format_non_date( + business_address.delivery_instructions)) + + self.__business_address_sheet_row_index += 1 + + def __write_filing_to_excel(self, business_identifier, filing): + self.__filing_sheet.write( + self.__filing_sheet_row_index, 0, format_non_date(business_identifier)) + + self.__filing_sheet.write( + self.__filing_sheet_row_index, 1, format_json(filing._filing_json)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 2, format_date(filing._completion_date)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 3, format_date(filing._filing_date)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 4, format_non_date(filing._filing_type)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 5, format_date(filing.effective_date)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 6, format_non_date(filing._payment_token)) + + self.__filing_sheet.write(self.__filing_sheet_row_index, 7, format_date( + filing._payment_completion_date)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 8, format_non_date(filing.colin_event_id)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 9, format_non_date(filing.status)) + self.__filing_sheet.write( + self.__filing_sheet_row_index, 10, format_boolean(filing.paper_only)) + + self.__filing_sheet_row_index += 1 diff --git a/legal-test-fixture/legal_test_api/api/converter/JsonConverter.py b/legal-test-fixture/legal_test_api/api/converter/JsonConverter.py new file mode 100644 index 0000000000..2f1725f62a --- /dev/null +++ b/legal-test-fixture/legal_test_api/api/converter/JsonConverter.py @@ -0,0 +1,122 @@ +from flask import json, jsonify +from legal_api.models.business import Business, Director, Address +from legal_api.models.office import Office, OfficeType +import logging +from legal_test_api.api.converter.utils import format_date, format_non_date, format_boolean, format_json + + +class JsonConverter(): + + def convert_to_json(self, business_list): + json_list = [] + + # convert each business to JSON + for business in business_list: + json_list.append(self.__json_business(business)) + + # return a single JSON object + return jsonify({'businesses': json_list}) + + def __json_business(self, business): + + offices = business.offices.all() + registeredOffice = self.__create_office_addresses(offices, OfficeType.REGISTERED) + recordsOffice = self.__create_office_addresses(offices, OfficeType.RECORDS) + + d = { + 'identifier': format_non_date(business.identifier), + 'legalName': format_non_date(business.legal_name), + 'legalType': format_non_date(business.legal_type), + 'foundingDate': format_date(business.founding_date), + 'dissolutionDate': format_date(business.dissolution_date), + 'lastAnnualReport': format_date(business.last_ar_date), + 'lastAnnualGeneralMeetingDate': format_date(business.last_agm_date), + 'fiscalYearEndDate': format_date(business.fiscal_year_end_date), + 'taxId': format_non_date(business.tax_id), + 'lastLedgerId': format_non_date(business.last_ledger_id), + 'lastRemoteLedgerId': format_non_date(business.last_remote_ledger_id), + 'lastLedgerTimestamp': format_date(business.last_ledger_timestamp), + 'submitterUserId': format_non_date(business.submitter_userid), + 'lastModified': format_date(business.last_modified), + 'directors': self.__json_directors(business.directors), + 'registeredOffice': registeredOffice, + OfficeType.REGISTERED: format_non_date(registeredOffice), + OfficeType.RECORDS: format_non_date(recordsOffice), + 'filings': self.__json_filings(business.filings) + } + + return d + + def __json_directors(self, directors): + directors_json_list = [] + + for director in directors: + d = { + 'firstName': format_non_date(director.first_name), + 'middleInitial': format_non_date(director.middle_initial), + 'lastName': format_non_date(director.last_name), + 'title': format_non_date(director.title), + 'appointmentDate': format_date(director.appointment_date), + 'cessationDate': format_date(director.cessation_date), + 'deliveryAddress': self.__format_address(director.delivery_address), + 'mailingAddress': self.__format_address(director.mailing_address) + } + directors_json_list.append(d) + + return directors_json_list + + def __format_address(self, value): + return_value = None + if value: + return_value = { + 'addressType': format_non_date(value.address_type), + 'street': format_non_date(value.street), + 'streetAdditional': format_non_date(value.street_additional), + 'city': format_non_date(value.city), + 'region': format_non_date(value.region), + 'country': format_non_date(value.country), + 'postalCode': format_non_date(value.postal_code) + } + return return_value + + def __json_filings(self, filings): + filings_json_list = [] + + for filing in filings: + d = { + 'completionDate': format_date(filing._completion_date), + 'filingDate': format_date(filing._filing_date), + 'filingType': format_non_date(filing._filing_type), + 'effectiveDate': format_date(filing.effective_date), + 'paymentToken': format_non_date(filing._payment_token), + 'paymentCompletionDate': format_date(filing._payment_completion_date), + 'colinEventId': format_non_date(filing.colin_event_id), + 'status': format_non_date(filing._status), + 'paperOnly': format_boolean(filing.paper_only), + # Don't want to use format_json here because we're + # running jsonify later and it will get all escaped + 'filingJson': format_non_date(filing._filing_json) + } + filings_json_list.append(d) + + return filings_json_list + + def __create_office_addresses(self, offices, office_type): + office_addresses = None + for office in offices: + if office_type == office.office_type: + mailing_address = None + delivery_address = None + + office_addresses = office.addresses.all() + for office_address in office_addresses: + if office_address.address_type == Address.MAILING: + mailing_address = office_address + elif office_address.address_type == Address.DELIVERY: + delivery_address = office_address + office_addresses = { + 'mailingAddress': self.__format_address(mailing_address), + 'deliveryAddress': self.__format_address(delivery_address) + } + return office_addresses + diff --git a/legal-test-fixture/legal_test_api/api/converter/utils.py b/legal-test-fixture/legal_test_api/api/converter/utils.py new file mode 100644 index 0000000000..2adb58fd60 --- /dev/null +++ b/legal-test-fixture/legal_test_api/api/converter/utils.py @@ -0,0 +1,49 @@ +from enum import Enum +from flask import json +import logging + + +def format_date(value): + return_value = None + if value: + return_value = str(value) + return return_value + + +def format_boolean(value): + return_value = None + if not value == None: + return_value = value + return return_value + + +def format_non_date(value): + return_value = None + if value: + return_value = value + return return_value + +# If we have a JSON value (like a filing) we can't save it as a JSON string because +# flask jsonify will escape everything + + +def format_json(value): + return_value = None + # for some reason sql_alchemy returns this as a list of strings? + # --> mystery solved: the app was doing loads before saving, so it didn't need to be loaded after + # if value and len(value) > 0: + # logging.warning(type(value)) + # return_value = json.loads(value[0]) + if value: + return_value = json.dumps(value) + return return_value + + +class SheetName(Enum): + """Render an Enum of the names of the sheets.""" + + BUSINESS = 'Businesses' + BUSINESS_ADDRESS = 'Business_Addresses' + DIRECTOR = 'Directors' + DIRECTOR_ADDRESS = 'Director_Addresses' + FILING = 'Filings' diff --git a/legal-test-fixture/legal_test_api/config.py b/legal-test-fixture/legal_test_api/config.py new file mode 100755 index 0000000000..82ee014863 --- /dev/null +++ b/legal-test-fixture/legal_test_api/config.py @@ -0,0 +1,6 @@ +import os + + +class Config: + SQLALCHEMY_TRACK_MODIFICATIONS = False + SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') diff --git a/legal-test-fixture/manage.py b/legal-test-fixture/manage.py new file mode 100755 index 0000000000..08c36cf169 --- /dev/null +++ b/legal-test-fixture/manage.py @@ -0,0 +1,9 @@ +import unittest +from flask.cli import FlaskGroup +from legal_api import create_app, db + +app = create_app() +cli = FlaskGroup(create_app=create_app) + +if __name__ == '__main__': + cli() diff --git a/legal-test-fixture/requirements.txt b/legal-test-fixture/requirements.txt new file mode 100755 index 0000000000..be8ea6d97f --- /dev/null +++ b/legal-test-fixture/requirements.txt @@ -0,0 +1,14 @@ +flask-cors==3.0.8 +xlrd==1.2.0 +xlwt==1.3.0 +Flask-SQLAlchemy==2.4.1 +Flask==1.1.1 +SQLAlchemy-Continuum==1.3.9 +SQLAlchemy-Utils==0.34.2 +SQLAlchemy==1.3.10 +datedelta==1.3 +flask-marshmallow==0.10.1 +marshmallow-sqlalchemy==0.19.0 +marshmallow==2.20.5 +psycopg2-binary==2.8.4 +git+https://github.com/bcgov/lear.git#egg=registry_schemas&subdirectory=schemas \ No newline at end of file diff --git a/legal-test-fixture/test/spreadsheets/businesses.xls b/legal-test-fixture/test/spreadsheets/businesses.xls new file mode 100644 index 0000000000..bbcdca1115 Binary files /dev/null and b/legal-test-fixture/test/spreadsheets/businesses.xls differ diff --git a/makefile b/makefile index fd98652a4c..b7eb278db0 100644 --- a/makefile +++ b/makefile @@ -13,6 +13,9 @@ local-project: setup-local-env build-local-project run-local-project ## Sets the configuration to a local-build setup-local-env: @cp ./coops-ui/public/config/local-configuration.json ./coops-ui/public/config/configuration.json + @cp -R ./legal-api/src/legal_api/models ./legal-test-fixture/legal_api + @cp -R ./legal-api/src/legal_api/exceptions ./legal-test-fixture/legal_api + @cp ./legal-api/src/legal_api/schemas.py ./legal-test-fixture/legal_api ## Builds the local project build-local-project: