diff --git a/README.md b/README.md index eaddc82..0bc73d1 100644 --- a/README.md +++ b/README.md @@ -1,153 +1,116 @@ -# Project Pine Beetle -Last Updated: 11.20.2019 -## Table of Contents -- Project Overview -- Project Architecture -- Developer Information -- Project Status -- Authors -- Acknowledgements +# Project Pine Beetle Backend + +This is a Node server for interacting with the Pine Beetle Prediction client application. + ## Project Overview + Project Pine Beetle is a web application that visualizes data on Southern Pine Beetle outbreaks in 16 states across the US. This tool uses a predictive model to predict future outbreaks and movements of Southern Pine Beetles. -On the front-end, this application provides valuable information for USFS researchers and state forest rangers to see information related to past outbreaks and predictions about future outbreaks. This application also provides information to the general public about threats facing their communities. +On the frontend, this application provides valuable information for USFS researchers and state forest rangers to see information related to past outbreaks and predictions about future outbreaks. This application also provides information to the general public about threats facing their communities. -On the back-end, this application aggregates data collected from USFS and state forest rangers on outbreaks and beetle counts, then uses those values to display historical data and future predictions. The predictive model used to generate predictions is written in R. All data is stored in a NoSQL database, allowing for easy pre and post-processing. Using an Express server, all calculations are made in JavaScript (outside of the predictive model), and all data is stored in JSON format. +On the backend, this application aggregates data collected from USFS and state forest rangers on outbreaks and beetle counts, then uses those values to display historical data and future predictions. The predictive model used to generate predictions is written in R. All data is stored in a MongoDB database, allowing for easy pre and post-processing. Using an Express server, all calculations are made in JavaScript (outside of the predictive model and Mongo summarization/aggregation algorithms), and all data is stored in JSON format. Project Pine Beetle is a collaboration between Professor Matt Ayres of Dartmouth College, Professor Carissa Aoki of Bates College, the United States Forest Service (USFS), and the Dartmouth Applied Learning and Innovation (DALI) Lab. -## Project Architecture -### Back-end -#### NoSQL Database: -Non-relational database built using MongoDB, responsible for storing data on pine beetle trappings and spots. The database was pre-populated with data collected from 1986-2010 which was stored locally and can be seen in the following [repository](https://github.com/dali-lab/pine-beetle-backend) in `src/data/`. - -Additionally, future data is imported to the database from Survey123, a data collection platform built on top of ArcGIS. All entries drawn from Survey123 come in on a trap-level. Data is collected across a 4-6 week trapping period, then aggregated from a trap level to a forest level. The data is pulled from Survey123 to the database as trapping data comes in. Seeing the surveys and data from Survey123 is strictly closed to the public. All data is publicly accessible on the front-end. - -#### Data: -Trapping data is entered in the spring. Spot data is entered in the fall. For information on variable names, see the back-end repository linked [here](https://github.com/dali-lab/pine-beetle-backend) in `src/models`. - -Between Spring 2017 and Spring 2018, the USFS shifted over from manual data collection (Microsoft Excel) to data collection through Survey123, an online platform built on top of ArcGIS. A representation of the manually collected data can be seen in `src/data/` on the back-end, as well as within the database (currently closed to the public). Survey123 data is visible both through the database and through the Survey123 platform. Information on querying data from Survey123 and ArcGIS can be found [here](https://developers.arcgis.com/rest/services-reference/query-feature-service-layer-.html). - -### Front-end -You can the front-end repository [here](https://github.com/dali-lab/pine-beetle-frontend). - -## Developer Information -### Installation: -#### Tools: -- You will need [Node.js](https://nodejs.org/en/), [yarn](https://yarnpkg.com/en/), [mongo/mongoDB](https://www.mongodb.com/), and [heroku](https://www.heroku.com) installed locally in order to build, run and develop this project. - -- Tool installation instructions (for mac, using homebrew) - - `brew install node` (you will need version >=9.x and <= 10.x) - - Note: for advanced usage, we also recommend installing Node.js via a version manager such as [nvm](https://github.com/creationix/nvm) instead of with homebrew. To do so, run `curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash`. Be sure to set your `.bash_profile` file by following the instructions listed in the [nvm repository](https://github.com/creationix/nvm). - - `brew install yarn` - - `brew install mongo` - - `brew install heroku/brew/heroku` - - `brew install python` - - `brew install R` - -#### Back-end repository: -- For the back-end, run `git clone https://github.com/dali-lab/pine-beetle-backend`. -- `cd pine-beetle-backend`. Then make sure you are on the master branch. -- Install necessary packages and dependencies with `yarn install`. -- Start a local server with `yarn start` -- Build with `yarn build` -- Deploy using heroku by setting a remote and running `git push heroku master` - -#### Contents -- `src/` contains the server and router -- `src/models` contains the data models for the database -- `src/controllers` contains the mongoose controllers -- `src/data` contains local copies of the initially imported data -- `src/importing-scripts` contains scripts used to manually upload data to the database as well as scripts for pulling Survey123 data. - -#### Exploring the DB -- Before you start anything, make sure mongo is installed correctly -- Run `mongod` -- Run `mongo` -- Run `use db` -- Explore database contents with calls such as `.find({})` - -#### Adding Data to DB -To locally import the csv file in `src/data/`: -`mongoimport --db pb-dev --collection spot-prediction --type csv --headerline --file ./src/data/sample_import_data.csv` - -#### Deployment -- Deploy to heroku by setting a remote: `heroku git:remote -a pine-beetle-prediction`. You must be signed into a Heroku account with access to this project to gain access. -- Then run `git push heroku master` to deploy and build. -- See [heroku deployment documentation](https://devcenter.heroku.com/articles/git) for more. - -## Project Status -As of November 19th, 2019, Project Pine Beetle will not be under active development. It is intended to be under active development again starting in Winter 2020. - -### Implemented: Fall, 2018 -- Database in MongoDB developed -- Historical data (1986-2010) uploaded to DB -- Pipeline constructed for uploading Survey123 data to DB -- Frontend framework developed -- Data retrieval from database implemented -- Data handling on front-end implemented -- User selection tools implemented -- Rudimentary data visualization in map form implemented -- Rudimentary data wrangling implemented -- Predictive model implemented to run in R through JavaScript -- Model updating process implemented - -### Implemented: Winter, 2019 -- Finalized data pipeline from Survey123 to MongoDB -- Created numerous routes on the back-end to query and serve data to the front-end -- Constructed capability to run R model using various feature inputs as well with a year/state/forest combination. -- Created routes for running the R model and returning results to front-end -- Converted front-end to React.js -- Built full historical data visualization -- Built full predictive model page -- Implemented fresh redesigns -- Constructed private capability for partners to pull Survey123 data to the database - -### Implemented: Fall, 2019 -- Simplified code -- Restructured data models -- Enforced MVC model in backend -- Ensured functional pipeline from Survey123 to MongoDB - -### Expected Implementation: Winter, 2020 -- Automate pipeline from Survey123 to MongoDB for updated previously seen data -- Improve model -- Improve run/load-times -- Cache/store previous model runs on server or in database to both improve run times and allow future versions of the predictive model to use previous model runs as inputs -- Implement a CDN -- Improve historical data visualization and predictive model visualization -- Add more educational and explanatory features/information for the general public to learn about this problem and this tool -- Implement additional features requested by the partners - - -### Future Directions -This product illustrates the threats facing communities in a visual manner. It is well suited to visualize any epidemic or spreading threat. It could be generalized and implemented for visualizing risk of forest fires, spread of disease, genetic diversity, or any threat that is predictable, has the potential to propagate outward, and displays a set of observable qualities indicating risk. Southern Pine Beetles may be just the beginning to the uses of a tool like this. - -## Team Members - -### Fall 2019 -- Amanda Bak, Project Manager -- Anuj Varma, Developer -- Emma Langfitt, Developer - -### Fall 2018 -- Thomas Monfre, Project Manager -- Madeline Hess, Developer -- Isabel Hurley, Developer - -### Winter 2019 -- Mo Zhu, Project Manager -- Thomas Monfre, Developer -- Madeline Hess, Developer -- Emi Hayakawa, Designer -- Bella Jacoby, Designer - -## README Authors -Amanda Bak, Anuj Varma, Emma Langfitt. - -## Acknowledgements -- This project was built in partnership with Professor Carissa Aoki of Bates College and Professor Matt Ayres of Dartmouth College. We thank them for approaching the DALI Lab and cooperating with us to build this product. -- We would like to thank many representatives from the US Forest Service and the Georgia Forestry Commission for their help, feedback and willingness to participate in user interviews. Particular thanks to Michael Torbett. -- Thank you to Tim Tregubov, Lorie Loeb, Natalie Jung, and Erica Lobel for their help, guidance, and advice. -- Shout out to Paula Mendoza for guiding us during the first part of this project. The direction of the final product is in large part the result of your positive, guiding influence. Thanks! -- Model for writing up this README drawn from https://github.com/dartmouth-cs98/18w-si32 +## Architecture + +This web server uses [Express.js](https://expressjs.com/) to set up basic routing. We use [mongoose](https://www.npmjs.com/package/mongoose) for connecting and writing to our database, which is hosted with [MongoDB Atlas](https://www.mongodb.com/). We use [bcrypt](https://www.npmjs.com/package/bcrypt) for assisting with authentication. + +We have two other repositories that comprise this application. Our [frontend](https://github.com/dali-lab/pine-beetle-frontend) is written with React and sends HTTP requests to this server to fetch data from the database and perform admin authentication actions. Our [automation server](https://github.com/dali-lab/pine-beetle-automation) is used for aggregating data from the USFS and restructuring it to our data format. Data comes from the USFS via several webhooks from [Survey123](https://survey123.arcgis.com/). See each of these repositories for more information. + +## Setup + +You must have [Node](https://nodejs.org) and [yarn](https://yarnpkg.com/) installed to run this project. + +1. Clone the repository +2. `yarn install` +3. Add a `.env` file and paste in the necessary contents (see Handoff Document for this) +4. `yarn start` to run in the local development environment + +## Repository Structure + +``` +src/ + constants/ [all constants and mapping files] + controllers/ [controllers for performing CRUD on the models] + middleware/ [middleware functions for each request] + models/ [all data models] + routers/ [all routers for exposing routes and calling controller functions] + utils/ [utility functions] + index.js [server setup file] +.babelrc [babel setup] +.eslintrc [eslint setup] +package.json [package] +``` + +### Data Flow + +Requests come in via the routers defined in `routers/`. Each router is hooked up to the main server file (`src/index.js`). Each request may be prefixed with a middleware function if authentication is required to access the route. + +This server only fetches data for each of the data collections (trapping, spot, and predictions). For user data and authentication, we perform reads and writes to the database. + +For user data, the routers invoke one or more controller functions declared in `controllers/`. Each of the controller functions then use a mongoose data model from `models/` to perform database actions. + +For the other database collections (not the user collection), the routers use the `queryFetch` utility function declared in `utils/` to perform a direct database read. This function takes in a collection name and optional filters, and retrieves the necessary data. + +In both cases, the router then sends the fetched data back to the client. Each response is formatted like the following: + +```json +{ + "status": , + "type": , + "data": , + "error": +} +``` + +### Authentication + +We use bycrypt for performing auth mechanisms. A user signs up with an email and a password. We then salt and hash the password and store it in the database in the `salted_password` field on the user object. We then generate a JWT (JavaScript Web Token) and send that back to the client. + +Authenticated requests send this JWT as a Bearer token in the authorization header to identify the user. We take this token and decode it to identify which user made the request. + +When users login, they send their email and password. We then salt and hash the password and make sure it matches the stored `salted_password` in the database for the user that has that email. If it matches, we generate a JWT and send it to the client. + +When users change their user data (including password), they must send their auth token (the JWT) in the authorization header. If the user sends an updated password, we re-salt and re-hash that new password and update the `salted_password` field. We always ensure the user is properly authenticated before performing these update actions. + +No request to the frontend ever sends the `salted_password` field. We also keep the auth secret and salt rounds stored in environment variables, so as to keep the authentication process secure. + +## Routes + +See all available routes and their documentation [here](./docs/ROUTES.md). + +## Code Style + +We use async/await for all asynchronous functions. + +## Deployment + +Continuous deployment is setup with Heroku. + +Merging a PR to the `dev` branch will trigger a new build in the dev environment. When the build passes, an update will be released at [https://pine-beetle-prediction-dev.herokuapp.com](https://pine-beetle-prediction-dev.herokuapp.com). + +Merging a PR to the `release` branch will trigger a new build in the production environment. When the build passes, an update will be released at [https://pine-beetle-prediction.herokuapp.com](https://pine-beetle-prediction.herokuapp.com). + +Pull requests should always be first merged into the `dev` branch so they are staged in the development environment. After smoke testing the changes in the development environment, developers can then choose to release those changes into production by generating a `DEV TO RELEASE` pull request from the `dev` branch to the `release` branch. One this single PR is merged into `release`, the changes will be built into the production environment and will be accessible at the production URL [https://pine-beetle-prediction.herokuapp.com](https://pine-beetle-prediction.herokuapp.com). + +### Database Environments + +When this server is running in the development environment, it connects to the development database in MongoDB Atlas. When this server is running in the production environment, it connects to the production database in MongoDB Atlas. + +This is determined by the `MONGODB_URI` environment variable that is set in Heroku. The dev app uses the development database URI and the production app likewise uses the production database URI. + +## Contributors + +- Thomas Monfre +- Jeff Liu +- Angela Zhang + +### Past Project Members + +- Nathan Schneider +- John McCambridge +- Madeline Hess +- Isabel Hurley +- Anuj Varma +- Emma Langfitt diff --git a/docs/PREDICTIONS-COUNTY.md b/docs/PREDICTIONS-COUNTY.md new file mode 100644 index 0000000..2a06870 --- /dev/null +++ b/docs/PREDICTIONS-COUNTY.md @@ -0,0 +1,20 @@ +# County Predictions Data Operations + +## `GET /county-prediction` + +Returns county predictions objects that match specified query params. + +User can supply query params for: + +- `cleridPerDay` +- `county` +- `spbPerDay` +- `state` +- `trapCount` +- `year` + +## `POST /county-prediction/query` + +Returns county predictions objects that match provided mongo query. + +User can supply mongo query in body. diff --git a/docs/PREDICTIONS-RD.md b/docs/PREDICTIONS-RD.md new file mode 100644 index 0000000..1b53973 --- /dev/null +++ b/docs/PREDICTIONS-RD.md @@ -0,0 +1,20 @@ +# Ranger District Predictions Data Operations + +## `GET /rd-prediction` + +Returns ranger district predictions objects that match specified query params. + +User can supply query params for: + +- `cleridPerDay` +- `rangerDistrict` +- `spbPerDay` +- `state` +- `trapCount` +- `year` + +## `POST /rd-prediction/query` + +Returns ranger district predictions objects that match provided mongo query. + +User can supply mongo query in body. diff --git a/docs/ROUTES.md b/docs/ROUTES.md index 6a8d470..d143728 100644 --- a/docs/ROUTES.md +++ b/docs/ROUTES.md @@ -1,7 +1,10 @@ # Route Documentation - [Healthcheck](./HEALTHCHECK.md) -- [Spot Data](./SPOTS.md) +- [Spot Data County](./SPOTS-COUNTY.md) +- [Spot Data Ranger District](./SPOTS-RD.md) +- [Predictions Data County](./PREDICTIONS-COUNTY.md) +- [Predictions Data Ranger District](./PREDICTIONS-RD.md) - [Summarized County Data](./SUMMARIZED-COUNTY.md) - [Summarized Ranger District Data](./SUMMARIZED-RD.md) - [Unsummarized Trapping Data](./UNSUMMARIZED.md) diff --git a/docs/SPOTS-COUNTY.md b/docs/SPOTS-COUNTY.md new file mode 100644 index 0000000..6c717ce --- /dev/null +++ b/docs/SPOTS-COUNTY.md @@ -0,0 +1,20 @@ +# County Spot Data Operations + +## `GET /spot-data-county` + +Returns county spot data objects that match specified query params. + +User can supply query params for: + +- `county` +- `fips` +- `hostAc` +- `spots` +- `state` +- `year` + +## `POST /spot-data-county/query` + +Returns county spot data objects that match provided mongo query. + +User can supply mongo query in body. diff --git a/docs/SPOTS-RD.md b/docs/SPOTS-RD.md new file mode 100644 index 0000000..c8140e4 --- /dev/null +++ b/docs/SPOTS-RD.md @@ -0,0 +1,18 @@ +# Ranger District Spot Data Operations + +## `GET /spot-data-rangerdistrict` + +Returns ranger district spot data objects that match specified query params. + +User can supply query params for: + +- `rangerDistrict` +- `spots` +- `state` +- `year` + +## `POST /spot-data-rangerdistrict/query` + +Returns ranger district spot data objects that match provided mongo query. + +User can supply mongo query in body. diff --git a/docs/SPOTS.md b/docs/SPOTS.md deleted file mode 100644 index 3523cd2..0000000 --- a/docs/SPOTS.md +++ /dev/null @@ -1,21 +0,0 @@ -# Spot Data Operations - -## `GET /spot-data` - -Returns spot data objects that match specified query params. - -User can supply query params for: - -- `county` -- `fips` -- `hostAc` -- `rangerDistrict` -- `spbCount` -- `state` -- `year` - -## `POST /spot-data/query` - -Returns spot data objects that match provided mongo query. - -User can supply mongo query in body. diff --git a/docs/USERS.md b/docs/USERS.md index 03b55b6..c9fe66c 100644 --- a/docs/USERS.md +++ b/docs/USERS.md @@ -33,3 +33,7 @@ Returns JWT auth token for user if salted and hashed version of password matches Expects authorization header with Bearer token. Returns 200 with no payload if auth header is valid. This route can be used on the automation server for verifying protected routes. + +## `GET /user/forgot-password/:email` + +Generates random new password for user with provided email (if a user object for that email exists). Sends email with this new password to the user's email. User can then check their email to get their new password and login.