Skip to content

Latest commit

 

History

History
203 lines (142 loc) · 7.96 KB

README.md

File metadata and controls

203 lines (142 loc) · 7.96 KB

AFSC GAP Viz

Visual analytics web application for AFSC GAP data that can be used without writing code. It uses the afscgap library underneath and enables users to continue in code or through a spreadsheet export.



Quickstart

No install required! The application is hosted for the public at https://app.pyafscgap.org but may be run locally as described below as well.



Installation

This application can be self-hosted or run locally. Due to the high disk usage required for this tool to operate efficiently (about 1GB local disk storage), it is not installed by default when running pip install afscgap. To install:

$ git clone https://github.com/SchmidtDSE/afscgap.git
$ cd afscgap/afscgapviz
$ pip install -r requirements.txt
$ bash load_deps.sh

Note that this application still requires a dataset as described below. For ease of development, webpack is not required.



Usage

This application is a Flask application that requires access to a database in order to work correctly.


Creating a sqlite database

The default usage leverages a sqlite database which is read only when used by the application. A pre-built database can be downloaded into the afscgap/afscgapviz directory like so:

$ cd afscgap/afscgapviz
$ wget https://pyafscgap.org/community/geohashes.zip
$ unzip geohashes.zip
$ rm geohashes.zip

If you want to build the dataset yourself, run the build_database script like so:

$ cd afscgap/afscgapviz
$ bash build_database.sh

Starting the application

Once the database is in place, one can start the web application with python afscgapviz.py. If you need to create a Flask app manually, it can be done like so:

import flask

import afscgapviz

app = flask.Flask(__name__)

afscgapviz.build_app(app)

This may be useful when running outside a development server.


Using a non-sqlite database

Any DB API 2.0 compliant database library may be used. For example, pg8000 can be used to leverage Postgres instead of sqlite like so:

import contextlib
import threading

import flask
import pg8000

import afscgapviz

pg8000.dbapi.paramstyle = 'qmark'

app = flask.Flask(__name__)


def get_connection_no_cache():
    return pg8000.dbapi.connect()


def build_get_connection():
    lock = threading.Lock()
    cache = {'cache': None}

    def get_connection():
        connection = cache['cache']

        # Test connection
        if connection:
            try:
                cursor = connection.cursor()
                cursor.execute('SELECT 1')
                cursor.fetchone()
                cursor.close()
            except:
                connection = None

        # Make connection if needed
        if connection is None:
            connection = get_connection_no_cache()

        # Save for next time
        cache['cache'] = connection

        return connection

    @contextlib.contextmanager
    def get_connection_wrapped():
        lock.acquire()
        try:
            yield get_connection()
        finally:
            lock.release()

    return get_connection_wrapped


afscgapviz.build_app(
    app,
    conn_generator_builder=build_get_connection
)

Note that the development of the tool is focused on sqlite. Pull requests are welcome for fixes for other backends but changes to SQL must be compatible with sqlite.



Data quality and limitations

Invalid data are excluded and only data with area swept in hectares, catch (count and weight in kg), and temperatures (both bottom and surface) are retained. No effort is made to try to "fix" records with incorrect data. Information is used as provided by the AFSC GAP API service. Furthermore, aggregation uses a single latitude / longitude point for a record and it remains possible that a haul may spill out of a geohash causing some approximation in catch per unit effort (CPUE) calculation. For more details on the "zero catch" inference methods used, see main README.



License

Like the rest of the the project, we are happy to make this library available under the BSD 3-Clause license. See LICENSE for more details. (c) 2023 Regents of University of California. See the Eric and Wendy Schmidt Center for Data Science and the Environment at UC Berkeley.



Local development

After installing dev dependencies (pip install -r requirements.txt), we recommend the following local checks:

$ nose2
$ mypy *.py
$ pyflakes *.py
$ pycodestyle *.py

Note these checks are run by CI / CD. Furthermore, JS tests can be run via grunt from the root directory or by:

$ python -m http.server

Then, direct your browser to http://0.0.0.0:8000/static/test/test.html.



Contributing

We invite contributions via our project Github. Please read the CONTRIBUTING.md file for more information.



Open source

In addition to using the afscgap library elsewhere in this repository, the following open source tools are used:

In CI / CD, this project also uses QUnit under the MIT License from OpenJS Foundation and other contributors as well as Grunt under an MIT-like License from jQuery Foundation and other contributors.

Thank you to all of these projects for their contribution. Note that CDNs are not used for privacy reasons.



Citations

The following are citations specifically for the visual analytics tool not covered in the open source section: