Merge pull request #689 from hackforla/684-BACK-PostmanIntegration

postman integration

.github/workflows/Continuous_Integration_Backend.yml

@@ -1,26 +1,34 @@
 name: Dev_CI_Backend
-on: [pull_request]
+
+on:
+  pull_request:
+    paths:
+      - 'server/**'
+
+defaults:
+  run:
+    shell: bash
+    working-directory: server
 
 jobs:
   build:
     runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version: [3.7]
     steps:
-    - uses: actions/checkout@v1
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v1
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Install dependencies
-      run: |
-        sudo apt install libpq-dev python3-dev
-        python -m pip install --upgrade pip
-        pip install -r server/api/requirements.txt
-        cp server/.env.example server/api/.env
-    - name: Lint with flake8
-      run: flake8 server
-    # disabled until we have a test DB to connect to
-    # - name: Test with pytest
-    #   run: pytest server
+      - uses: actions/checkout@v2
+
+      - name: Install and Run Api
+        run: |
+          cp .env.example .env
+          echo SOCRATA_TOKEN=${{ secrets.SOCRATA_TOKEN }} >> .env
+          docker-compose up --no-start api
+          docker-compose run api python bin/db_seed.py --years 2020 --rows 500 --reset
+          docker-compose up -d api
+
+      - name: Linting
+        run: docker-compose run api flake8
+
+      - name: Unit Tests
+        run: docker-compose run api pytest
+
+      - name: Postman Tests
+        run: chmod +x postman/test.sh && postman/test.sh

server/.env.example

@@ -51,3 +51,5 @@ DB_NAME=311_db
 DB_HOST_PORT=5433
 API_HOST_PORT=5000
 API_RESTART_POLICY=no
+
+################################# OVERRIDES ##############################

server/README.md

@@ -59,7 +59,7 @@ If you decide later that you need more data, just run the command again with the
 ### Optional Dependencies
 
   - #### Postman
-
+[Postman](https://www.postman.com/) is an api-development tool that lets you save api requests so you can run them over and over without having to remember what all the parameters are (or use the awkward syntax of `curl`). You can also group api calls together in collections, and run all of the api calls in a collection at once -- which is a great way to test the entire api. If you'd like to use it, see the README in `/server/postman`, which explains how to set it up with a collection that contains pre-defined api calls for all of our endpoints.
 
 ### Useful commands
 ```
@@ -68,9 +68,7 @@ docker-compose up --build             # start the backend services after rebuild
 
 docker-compose run api bash           # log in to api shell
 docker-compose run api flake8         # lint your python code
-docker-compose run api pytest         # run unit tests against python code
-
-docker-compose run redis redis-cli    # run the redis cli  
+docker-compose run api pytest         # run unit tests against python code 
 ```
 
 ### Using the python interpreter

server/api/bin/db_seed.py

@@ -28,6 +28,11 @@ def get_cli_args():
         help='number of rows per call to Socrata api ' +
              f'(default: {Socrata.BATCH_SIZE})')
 
+    parser.add_argument(
+        '--reset',
+        action='store_true',
+        help='reset the database before seeding')
+
     return parser.parse_args()
 
 
@@ -58,5 +63,9 @@ def parse_years(years):
     years = parse_years(args.years)
     rows = -1 if args.rows is None else args.rows
     batch = Socrata.BATCH_SIZE if args.batch is None else args.batch
+    reset = args.reset
+
+    if reset:
+        db.reset()
 
     db.requests.add_years(years, rows_per_year=rows, batch_size=batch)

server/api/requirements.txt

@@ -28,15 +28,17 @@ pysupercluster==0.7.6
 pytest==5.3.3
 python-dateutil==2.8.1
 python-dotenv==0.13.0
+python-http-client==3.2.7
 pytz==2019.3
 redis==3.5.0
 requests==2.23.0
 requests-async==0.5.0
 rfc3986==1.3.2
 sanic==19.9.0
+sanic-compress==0.1.1
 Sanic-Cors==0.10.0.post3
-sanic_compress==0.1.1
 Sanic-Plugins-Framework==0.9.2
+sendgrid==6.3.1
 six==1.14.0
 sodapy==2.0.0
 SQLAlchemy==1.3.13

server/api/src/app.py

@@ -20,10 +20,10 @@
         ['GET', 'HEAD'], R.status.api),
 
     '/status/sys': (
-        ['GET', 'HEAD'], R.status.sys),
+        ['GET'], R.status.sys),
 
     '/status/db': (
-        ['GET', 'HEAD'], R.status.db),
+        ['GET'], R.status.db),
 
     '/servicerequest/<srnumber>': (
         ['GET'], R.request_detail),

server/api/src/services/github.py

@@ -44,13 +44,13 @@ async def create_issue(title,
             issue_number = response_content['number']
             return issue_id, issue_number
         except requests.exceptions.HTTPError as errh:
-            return errh
+            raise errh
         except requests.exceptions.ConnectionError as errc:
-            return errc
+            raise errc
         except requests.exceptions.Timeout as errt:
-            return errt
+            raise errt
         except requests.exceptions.RequestException as err:
-            return err
+            raise err
 
 
 async def add_issue_to_project(issue_id, content_type='Issue'):
@@ -80,10 +80,10 @@ async def add_issue_to_project(issue_id, content_type='Issue'):
             response.raise_for_status()
             return response.status_code
         except requests.exceptions.HTTPError as errh:
-            return errh
+            raise errh
         except requests.exceptions.ConnectionError as errc:
-            return errc
+            raise errc
         except requests.exceptions.Timeout as errt:
-            return errt
+            raise errt
         except requests.exceptions.RequestException as err:
-            return err
+            raise err

server/api/test/test_app.py

@@ -0,0 +1,5 @@
+import app
+
+
+def test_app():
+    assert hasattr(app, 'start')

server/api/test/test_db.py

@@ -0,0 +1,16 @@
+import db
+from datetime import datetime
+
+
+class TestDB:
+    def test_version(self):
+        version = db.version()
+        assert isinstance(version, int)
+
+    def test_last_updated(self):
+        last_updated = db.info.last_updated()
+        assert isinstance(last_updated, datetime)
+
+    def test_rows(self):
+        rows = db.info.rows()
+        assert isinstance(rows, dict)

server/postman/README.md

@@ -0,0 +1,44 @@
+## Setup
+
+### 1. Install and run Postman
+
+Download it [here](https://www.postman.com/downloads/) and start it up.
+
+### 2. Create a 311 workspace (optional)
+
+If you create a workspace, all of your 311 stuff will be grouped together, so it won't get jumbled up with anything else you're doing in Postman.
+
+To create one, select "create new" from dropdown menu in the middle of the nav bar (top of the window). Give it a name, select "Personal" type (instead of "Team"), and hit "Create Workspace".
+
+### 3. Import this folder
+
+Click the Import button at the top left, select "Folder" from the tabs, and then select this folder (`/server/postman`) from the finder. The import will add two collections to Postman:
+  - **311-all**: a collection containing all of the endpoints that the api currently supports, with prepopulated params for each api call.
+  - **311-CI**: a collection containing all of the tests we run during continuous integration. These include tests for bad input -- e.g., a missing required param should return 400, an unsupported endpoint should return a 404.
+
+The import will also add two environments:
+  - **311-local**: all api calls go to your local server (which should be running)
+  - **311-prod**: api calls go to the production server
+
+### 4. Activate the `311-local` environment.
+
+Just select it from the dropdown at the top right of the window.
+
+## Usage
+
+### Run a single api call
+
+Open the `311-all` collection on the left, and click one of the endpoints. Then click the blue "Send" button next to the url bar. This will send a request to the endpoint and show you the result. If you click around underneath the url bar, you'll find the body of the request (if it's a POST), and in the section below that, the body of the response.
+
+### Run an entire collection
+
+Click "Runner" at the top left to open the collection runner. Then select one of the two collections, select the `311-local` environment, and hit the blue button. Postman will make an api call to your local server for each item in the collection, and print a summary of the results.
+
+Note that the `311-CI` collection contains all of the tests we run whenever you submit a pull request to `dev`. So if you want to help make sure your PR gets approved, run these tests before submitting.
+
+### Run CI tests from the command line
+
+As an alternative to running the CI tests in Postman, you can run them from the command line. Run this command from the `/server` directory:
+```
+chmod +x postman/test.sh && postman/test.sh
+```