Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V 1.0.0 #282

Merged
merged 70 commits into from
Dec 21, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
ecf24f8
add search for every text anything
bitner Jul 1, 2020
f50b94d
add elastic
bitner Jul 27, 2020
d2bd01c
Merge remote-tracking branch 'origin/searcheverything' into elastic
bitner Jul 27, 2020
ead5c97
Add startserver redirects, set unicode supporting fonts in tex template
jvntf Jul 28, 2020
a825ce9
Remove sanity check line
jvntf Jul 28, 2020
5fd1db3
Update startserver.sh
jvntf Jul 29, 2020
b6034f2
Add placement param to figures
jvntf Jul 29, 2020
97c4af2
elastic search
bitner Jul 30, 2020
1c26b07
cleanup
bitner Jul 30, 2020
f75a547
Use H placement param and float package
jvntf Jul 31, 2020
ef34890
Merge pull request #266 from developmentseed/fix/257-image-placement
jvntf Jul 31, 2020
e2acb59
Change 0 to /dev/null
jvntf Aug 3, 2020
e7a2a4e
add contact triggers, add search proxy to fastapi
bitner Aug 3, 2020
eaab347
remove unused import
bitner Aug 3, 2020
67fd157
Add documentation of font choice
jvntf Aug 4, 2020
1fc93f4
Resolve conflicts
jvntf Aug 4, 2020
4b7061e
add initial call to /reindex as part of startserver.sh
bitner Aug 4, 2020
57b52b1
Add font documentation to readme
jvntf Aug 5, 2020
8bc79d8
Merge pull request #264 from developmentseed/fix/176-support-unicode
jvntf Aug 5, 2020
d840673
Merge branch 'develop' into elastic
jvntf Aug 11, 2020
14b62fa
Add CORS defns
jvntf Aug 11, 2020
77a5f8b
add cloudformation (not working yet) mv pdf->fastapi
bitner Sep 7, 2020
64c1a98
Merge branch 'elastic' of github.com:developmentseed/nasa-apt into el…
bitner Sep 7, 2020
df7408e
working elastic with cloudformation on aws
bitner Sep 21, 2020
1237ade
initial saml work
bitner Oct 2, 2020
26ef984
fix auth issue
bitner Oct 2, 2020
c43754b
fix for proxy issue with elastic search
bitner Oct 13, 2020
6dcd224
add env var for frontend url to use for cors
bitner Oct 21, 2020
8047787
add sync script for moving over database
bitner Oct 26, 2020
42ff15f
saml auth
bitner Oct 28, 2020
f23fa26
Merge branch 'elastic' of github.com:developmentseed/nasa-apt into saml
bitner Oct 28, 2020
919a1d2
initial somewhat working cloudformation for saml
bitner Oct 28, 2020
7870e6f
added new endpoint atbd journal type document generation
leothomas Nov 25, 2020
1afcfa9
added separate name for journal atbds
leothomas Nov 25, 2020
801a56e
Revert "added separate name for journal atbds"
leothomas Nov 25, 2020
3868163
Add database fields for journal details section
danielfdsilva Dec 3, 2020
b620bb7
updated algorithm_input_variables to use WYSIWYG element instead of text
leothomas Dec 3, 2020
311a575
finalized first pass of journal atbds
leothomas Dec 3, 2020
4be5623
Merge pull request #277 from developmentseed/feature/journal-db-fields
danielfdsilva Dec 4, 2020
d34bbcb
Include fixture data for atbd
danielfdsilva Dec 4, 2020
c7983db
fixed NaN value in tables
leothomas Dec 7, 2020
dfbc956
removed unused temp file
leothomas Dec 7, 2020
84fa077
WIP - working on docker-networking issue
leothomas Dec 8, 2020
22e112a
WIP - working on container networking, fixed journal acknowledgements…
leothomas Dec 8, 2020
164d145
cleaned up docker
leothomas Dec 8, 2020
65d2513
removed cache bypass
leothomas Dec 8, 2020
0b43d97
saml auth updates
bitner Dec 10, 2020
a63da1b
update .env.sample with saml conf
bitner Dec 10, 2020
ce347cb
add mock option for authentication
bitner Dec 10, 2020
c8cac1d
adjustments to cloud formation load balancer targets
bitner Dec 10, 2020
be36eb7
add documentation to syncdb.sh
bitner Dec 10, 2020
4440898
add documentation .env.sample
bitner Dec 10, 2020
f4cfb0c
add readme for deploying to https
bitner Dec 10, 2020
27f059c
updates to syncdb.sh to run migrations first
bitner Dec 10, 2020
e0544b7
added `align` wrapping around sample equation in full atbd and enable…
leothomas Dec 11, 2020
4335883
WIP
leothomas Dec 15, 2020
fe0319c
fixed `NaN` in table header, fixed table width and missing table capt…
leothomas Dec 16, 2020
c166327
fix return to when using mock auth
bitner Dec 16, 2020
218bd8f
add env variable for jwt to docker compose
bitner Dec 16, 2020
37ed8dc
update env var for anonymous user
bitner Dec 16, 2020
4daf9b2
Fix logout option
danielfdsilva Dec 16, 2020
c401100
Fix cloudformation variables
danielfdsilva Dec 16, 2020
3ef498a
Merge pull request #281 from developmentseed/saml
danielfdsilva Dec 16, 2020
58e7ca3
Merge branch 'develop' into journal
danielfdsilva Dec 16, 2020
1a2abd6
WIP
leothomas Dec 17, 2020
223ae52
Added JWT token to get request for logged in user when generating atb…
leothomas Dec 17, 2020
47008d6
Merge pull request #280 from developmentseed/feature/journal-pdf-endp…
danielfdsilva Dec 17, 2020
12c4da1
Fix cloudformation deploy
danielfdsilva Dec 18, 2020
83f29ba
Prep v1 release
danielfdsilva Dec 21, 2020
02a6858
Remove circle ci badge
danielfdsilva Dec 21, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion .env.sample
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
STACKNAME=nasa-apt-dev

# s3 endpoint (for local dev environment)
S3_ENDPOINT=http://localstack:4572
S3_ENDPOINT=http://localstack:4566

# s3 bucket name for pdf serialization cache. used by cloudformation/deploy.sh and by docker-compose.yml
PDFS_S3_BUCKET=nasa-apt-dev-pdfs
Expand All @@ -15,3 +15,23 @@ FIGURES_S3_BUCKET=nasa-apt-dev-figures

# postgrest api
REST_API_ENDPOINT=http://rest-api:3000

# url for database
DBURL=postgres://masteruser:password@db:5432/nasadb

# url for elasticsearch
ELASTICURL=http://localhost:9200

# url for FastApi
FASTAPI_HOST=http://localhost:8000

# Frontend Url used to set CORS origin for FastAPI
APT_FRONTEND_URL=http://nasa-apt-eltest-application.s3-website-us-east-1.amazonaws.com

# url for SAML IDP metadata
# IDP_METADATA_URL=https://dbspatial.us.auth0.com/samlp/metadata/vEJRZ0X6OtSMzkhBqIUMM62KDJhSNyG3
# IDP_METADATA_URL=https://auth.launchpad-sbx.nasa.gov/unauth/metadata/launchpad-sbx.idp.xml
# IDP_METADATA_URL=mock # Setting IDP_METADATA_URL=mock will bypass use of an idp server and will allow anyone to log in by going to /saml/sso

# Note: If the JWT_SECRET isn't sufficiently long, postgrest may complain
JWT_SECRET=lksdjlkasjasdlkjasdlkjfdlasdfkkllkjasdfhlksdjlkasdjasdlkjasdlkjsdflkjwelkwejrlkrjwlkwejlwekjrwelkrjewlkrjwelkrj
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ ecs/tex/Misc Test Files
venv
.env
.resources
.vscode
46 changes: 23 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,21 @@
# nasa-apt

**Version:** 0.3.0
**Version:** 1.0.0

## Local development
Code and issues relevant to the NASA APT project

[![CircleCI](https://circleci.com/gh/developmentseed/nasa-apt/tree/develop.svg?style=svg&circle-token=ffc901ab7ce00ffa5cef07cce59ff64a2c635d2b)](https://circleci.com/gh/developmentseed/nasa-apt/tree/develop)

The project API is built using [Postgrest](https://github.com/PostgREST/postgrest).

The startserver script uses `docker-compose` to build and run the development environment and
sample database:
sample database:

```shell script
./startserver.sh
```

This will create a complete development environment with an instance of the DB, the REST API, `localstack` for s3, and
the PDF serialization service.
the FastAPI Search / PDF serialization service.

- The Swagger API documentation is accessible via [http://localhost:8080](http://localhost:8080).
- The REST API is accessible via [http://localhost:3000](http://localhost:3000).
Expand Down Expand Up @@ -58,29 +56,28 @@ cd cloudformation
```

You will be prompted for a stack name and a master db password. The current
stacks are `nasa-aptv2-staging` and `nasa-aptv2-production`.
stack is `nasa-apt-v5-prod`

After the stack has been successfully deployed you can create the database tables.
You will need an installation of the `psql` command line client.
You will need an installation of the `psql` command line client.

You will also need to update the RDS instance's security policy to allow inbound traffic from the IP address of the machine where you
You will also need to update the RDS instance's security policy to allow inbound traffic from the IP address of the machine where you
are executing the deployment. (see Resources | DBInstance | Security and Network | Security Groups |
Edit inbound rules | Custom TCP, Port 5432, My IP).
Edit inbound rules | Custom TCP, Port 5432, My IP).

To create the schema and tables in the AWS RDS from the project root run
```shell script
cd db
./sqitch deploy --verify db:pg://{yourmasteruser}:{yourmasterpassword}@{yourRDSendpoint}:5432/nasadb
```

Because of PostgREST's schema reloading [model](http://postgrest.org/en/v5.2/admin.html#schema-reloading) some
underlying database changes may require a forced redeployment of the PostgREST ECS service to reflect the changes. (See Note in
Because of PostgREST's schema reloading [model](http://postgrest.org/en/v5.2/admin.html#schema-reloading) some
underlying database changes may require a forced redeployment of the PostgREST ECS service to reflect the changes. (See Note in
[Environments](#environments))

## Environments
There are currently 2 environments defined for NASA-APT, which follow specific branches
- Staging (`develop`): http://nasa-Publi-1UDVJHRLIQD2G-1353740340.us-east-1.elb.amazonaws.com
- Production (`master`): http://nasa-Publi-1LGW8ZYHL7SF7-1834206210.us-east-1.elb.amazonaws.com
- Production (`master`): https://apt.ds.io

**Given that deployment is a manual process it is important that the environments are kept up to date after a merge to `master` or `develop`.**

Expand All @@ -94,20 +91,20 @@ Steps to deploy:

```shell script
aws ecs update-service --force-new-deployment --cluster <cluster-id> --service <service-arn>
# e.g.
aws ecs update-service --force-new-deployment --cluster stackname-ECSCluster-nWSsDVGj9NXS --service stackname-svc-pgr
# e.g.
aws ecs update-service --force-new-deployment --cluster stackname-ECSCluster-nWSsDVGj9NXS --service stackname-svc-pgr
# then wait until the service's desired count == the running count (this will take about 10 minutes)
```

## Updating the PDF service
The PDF generation service uses docker and it is stored on amazon ECR. During the first cloudformation deployment, the container is created and uploaded, but subsequent updates need to be performed manually.
We're currently using a single ECR repo (nasa-apt/prod/pdf) to store the container and it is shared between the production and staging environments.
## Updating the FastAPI (PDF/Search) service
The PDF generation service uses docker and it is stored on amazon ECR. During the first cloudformation deployment, the container is created and uploaded, but subsequent updates need to be performed manually.
We're currently using a single ECR repo (nasa-apt/prod/fastapi) to store the container and it is shared between the production and staging environments.

1) Build the container
```
cd nasa-apt/pdf/
# from the pdf/Readme
docker build --target prod . -t nasa-apt/prod/pdf
cd nasa-apt/fastapi/
# from the fastapi/Readme
docker build --target prod . -t nasa-apt/prod/fastapi
```
2) Go to the [ECR page](https://us-east-1.console.aws.amazon.com/ecr/repositories?region=us-east-1), select the correct repo and click "View Push Commands".
3) Follow steps 1, 3, and 4.
Expand All @@ -116,13 +113,16 @@ docker build --target prod . -t nasa-apt/prod/pdf
aws ecs update-service --force-new-deployment --cluster <cluster> --service <service>
```

## Notes
The PDF serialization service supports unicode characters in text mode. The service uses the font `Latin Modern Math` which has a good coverage of unicode math symbols. See a list of symobls here: https://ctan.math.illinois.edu/macros/latex/contrib/unicode-math/unimath-symbols.pdf
A symbol which is not covered by the font will be rendered as a blank space. Unicode characters used in LaTeX math mode will not be rendered.

## Releases

**A new release should be created every time there's a merge to master.**

Releases are tied to a version number and created manually using GH's releases page.
The version in this README should be increased according to [semver](https://semver.org/) and the release tag should follow the format `v<major>.<minor>.<patch>`, ex: `v2.0.1`.
Releases are tied to a version number and created manually using GH's releases page.
The version in this README should be increased according to [semver](https://semver.org/) and the release tag should follow the format `v<major>.<minor>.<patch>`, ex: `v2.0.1`.
The release description should have a [changelog](https://gist.github.com/vgeorge/e6fd828987b2f7d62a447df2bd132c4a) with "Features", "Improvements" and "Fixes".


Expand Down
18 changes: 18 additions & 0 deletions cloudformation/README.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Instructions for changing a Load Balancer to use HTTPS

1) Deploy the stack using Cloudformation
2) Find the Load Balancer for the stack at https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LoadBalancers:sort=loadBalancerName
3) Select the checkbox for the Load Balancer and check on the Listeners tab in the box at the bottom
4) Click "Add listener"
5) Select HTTPS 443
6) Select "+ Add action" -> Forward to -> <stackname>-dummyTarget
7) Select "From ACM" and "apt.ds.io - ..." under Default SSL Certificate
8) Hit Save in the upper right and then the Back arrow next to Listeners in the upper left
9) Back at the bottom, select view/edit rules under the entry you just created
10) Click the + button then "+ Insert Rule"
11) Add the following rules making sure they end up in this order:
- IF Path is /saml/* THEN Forward to <stackname>-tg-fastapi
- IF Path is /fastapi/* THEN Forward to <stackname>-tg-fastapi
- IF Path is /* THEN Forward to <stackname>-tg-pgr
12) Go back to the list of Listeners and then to the HTTP 80 view/edit rules Remove all but the dummy rule
13) Have Olaf or someone with Route53 Permissions move apt.ds.io (or setup a new domain) to point to that load balancer
Loading