This is a tool within the Lintol system.
We recommend pipenv, but any setuptools compatible install method should work.
pipenv shell
python3 -m pip install -r requirements.txt
Or, for development,
pipenv shell
python3 -m pip install -r requirements-development.txt
python3 setup.py develop
From the project root directory.
ltldoorstep process tests/examples/data/protected_wrecks.geojson src/ltldoorstep_examples/boundary_checker_impr.py -e dask.threaded
python3 -m pytest tests
A processor is a function that runs a specific check or validation against data. Several examples are included within the doorstep repository, in the `ltldoorstep_examples` module - these are:
This takes in JSON files and checks all the points are within a particular boundary. You can test it against test/examples/data/protected_wrecks.geojson.
This takes CSV data and checks the data records are valid - ie, no duplicates, valid IDs.
Can be tested against test/examples/data/bad.csv.
This outputs the CSV file. Can be tested against test/examples/data/dispensing-by-contractor-june-2018.csv
A processor to categorise the file based on the location information.
To test use ltldoorstep process tests/examples/data/register-countries.json src/ltldoorstep_examples/dt_classify_location.py -e dask.threaded
Outputs data in a tabular format. Run the test for this using tests/examples/data/awful.csv
Processor to find any personally identifiable information within the data. This can be tested using test/examples/data/pii.csv
The data is checked against the open data the government has for location information on gov.uk. Can be tested using tests/examples/data/register-statistical-geography.csv
The data is checked to see if it is specific to Northern Ireland. This can be tested against test/examples/data/ni-hpi-by-property-type_package.json