Skip to content

Latest commit

 

History

History
380 lines (258 loc) · 17.9 KB

README.md

File metadata and controls

380 lines (258 loc) · 17.9 KB
Raw

Development

========================

If you're new to Django, see Getting Started with Django for an introduction to the framework.

Local Setup

  • Install Docker https://docs.docker.com/get-docker/

  • Initialize the application:

    cd src
docker-compose build

  • Run the server: docker-compose up

    Press Ctrl-c when you'd like to exit or pass -d to run in detached mode.

Visit the running application at http://localhost:8080.

Troubleshooting

  • If you are using Windows, you may need to change your line endings. If not, you may not be able to run manage.py.
  • Unix based operating systems (like macOS or Linux) handle line separators differently than Windows does. This can break bash scripts in particular. In the case of manage.py, it uses #!/usr/bin/env python to access the Python executable. Since the script is still thinking in terms of unix line seperators, it may look for the executable python\r rather than python (since Windows cannot read the carriage return on its own) - thus leading to the error usr/bin/env: 'python\r' no such file or directory
  • If you'd rather not change this globally, add a .gitattributes file in the project root with * text eol=lf as the text content, and refresh the repo
  • If you are using a Mac with a M1 chip, and see this error The chromium binary is not available for arm64. or an error involving puppeteer, try adding this line below into your .bashrc or .zshrc.
export DOCKER_DEFAULT_PLATFORM=linux/amd64

When completed, don't forget to rerun docker-compose up!

Branch Conventions

We use the branch convention of initials/branch-topic (ex: lmm/fix-footer). This allows for automated deployment to a developer sandbox namespaced to the initials.

Merging and PRs

History preservation and merge contexts are more important to us than a clean and linear history, so we will merge instead of rebasing. To bring your feature branch up-to-date wih main:

git checkout main
git pull
git checkout <feature-branch>
git merge orgin/main
git push

Resources:

Setting Vars

Non-secret environment variables for local development are set in src/docker-compose.yml.

Secrets (for example, if you'd like to have a working Login.gov authentication) go in .env in src/ with contents like this:

DJANGO_SECRET_LOGIN_KEY="<...>"

You'll need to create the .env file yourself. Get started by running:

cd src
cp ./.env-example .env

Get the secrets from Cloud.gov by running cf env getgov-YOURSANDBOX. More information is available in rotate_application_secrets.md.

Getting access to /admin on all development sandboxes (also referred to as "adding to fixtures")

The endpoint /admin can be used to view and manage site content, including but not limited to user information and the list of current applications in the database. However, access to this is limited to analysts and full-access users with regular domain requestors and domain managers not being able to see this page.

While on production (the sandbox referred to as stable), an existing analyst or full-access user typically grants access /admin as part of onboarding (see these instructions), doing this for all development sandboxes is very time consuming. Instead, to get access to /admin on all development sandboxes and when developing code locally, refer to the following sections depending on what level of user access you desire.

Adding full-access user to /admin

To get access to /admin on every non-production sandbox and to use /admin in local development, do the following:

  1. Login via login.gov
  2. Go to the home page and make sure you can see the part where you can submit a domain request
  3. Go to /admin and it will tell you that your UUID is not authorized (it shows a very long string, this is your UUID). Copy that UUID for use in 4.
  4. (Designers) Message in #getgov-dev that you need access to admin as a superuser and send them this UUID along with your desired email address. Please see the "Adding an Analyst to /admin" section below to complete similiar steps if you also desire an analyst user account. Engineers will handle the remaining steps for designers, stop here.

(Engineers) In src/registrar/fixtures_users.py add to the ADMINS list in that file by adding your UUID as your username along with your first and last name. See below:

 ADMINS = [
        {
            "username": "<UUID here>",
            "first_name": "",
            "last_name": "",
            "email": "",
        },
        ...
 ]
  1. (Engineers) In the browser, navigate to /admin. To verify that all is working correctly, under "domain requests" you should see fake domains with various fake statuses.
  2. (Engineers) Add an optional email key/value pair

Adding an analyst-level user to /admin

Analysts are a variant of the admin role with limited permissions. The process for adding an Analyst is much the same as adding an admin:

  1. Login via login.gov (if you already exist as an admin, you will need to create a separate login.gov account for this: i.e. [email protected])

  2. Go to the home page and make sure you can see the part where you can submit a domain request

  3. Go to /admin and it will tell you that UUID is not authorized, copy that UUID for use in 4 (this will be a different UUID than the one obtained from creating an admin)

  4. (Designers) Message in #getgov-dev that you need access to admin as a superuser and send them this UUID along with your desired email address. Engineers will handle the remaining steps for designers, stop here.

  5. (Engineers) In src/registrar/fixtures_users.py add to the STAFF list in that file by adding your UUID as your username along with your first and last name. See below:

 STAFF = [
        {
            "username": "<UUID here>",
            "first_name": "",
            "last_name": "",
            "email": "",
        },
        ...
 ]
  1. (Engineers) In the browser, navigate to /admin. To verify that all is working correctly, verify that you can only see a sub-section of the modules and some are set to view-only.
  2. (Engineers) Add an optional email key/value pair

Do note that if you wish to have both an analyst and admin account, append -Analyst to your first and last name, or use a completely different first/last name to avoid confusion. Example: Bob-Analyst

Adding an email address to the email whitelist (sandboxes only)

On all non-production environments, we use an email whitelist table (called Allowed emails). This whitelist is not case sensitive, and it provides an inclusion for +1 emails (like [email protected]). The content after the + can be any digit. The whitelist checks for the "base" email (example.person) so even if you only have the +1 email defined, an email will still be sent assuming that it follows those conventions.

To add yourself to this, you can go about it in three ways.

Permanent (all sandboxes):

  1. In src/registrar/fixtures_users.py, add the "email" field to your user in either the ADMIN or STAFF table.
  2. In src/registrar/fixtures_users.py, add the desired email address to the ADDITIONAL_ALLOWED_EMAILS list. This route is suggested for product.

Sandbox specific (wiped when the db is reset): 3. Create a new record on the Allowed emails table with your email address. This can be done through django admin.

More detailed instructions regarding #3 can be found here.

Adding to CODEOWNERS (optional)

The CODEOWNERS file sets the tagged individuals as default reviewers on any Pull Request that changes files that they are marked as owners of.

  1. Go to .github\CODEOWNERS

  2. Following the CODEOWNERS documentation, add yourself as owner to files that you wish to be automatically requested as reviewer for.

    For example, if you wish to add yourself as a default reviewer for all pull requests, add your GitHub username to the same line as the * designator:

    - * @abroddrick
+ * @abroddrick @YourGitHubUser

  3. Create a pull request to finalize your changes

Viewing Logs

If you run via docker-compose up, you'll see the logs in your terminal.

If you run via docker-compose up -d, you can get logs with docker-compose logs -f.

You can change the logging verbosity, if needed. Do a web search for "django log level".

Mock data

load.py called from docker-compose (locally) and reset-db.yml (upper) loads the fixtures from fixtures_user.py and the rest of the data-loading fixtures in that fixtures folder, giving you some test data to play with while developing.

See the database-access README for information on how to pull data to update these fixtures.

Running tests

Crash course on Docker's run vs exec: in order to run the tests inside of a container, a container must be running. If you already have a container running, you can use exec. If you do not, you can use run, which will attempt to start one.

To get a container running:

cd src
docker-compose build
docker-compose up -d

Django's test suite:

docker-compose exec app ./manage.py test

OR

docker-compose exec app python -Wa ./manage.py test  # view deprecation warnings

Linters:

docker-compose exec app ./manage.py lint

Testing behind logged in pages

To test behind logged in pages with external tools, like pa11y-ci or OWASP Zap, add

"registrar.tests.common.MockUserLogin"

to MIDDLEWARE in settings.py. Remove it when you are finished testing.

Reducing console noise in tests

Some tests, particularly when using Django's test client, will print errors.

These errors do not indicate test failure, but can make the output hard to read.

To silence them, we have a helper function less_console_noise:

from .common import less_console_noise
...
        with less_console_noise():
            # <test code goes here>

Or alternatively, if you prefer using a decorator, just use:

from .common import less_console_noise_decorator

@less_console_noise_decorator
def some_function():
  # <test code goes here>

Accessibility Testing in the browser

We use the ANDI browser extension from ssa.gov for accessibility testing outside the pipeline.

ANDI will get blocked by our CSP settings, so you will need to install the Disable Content-Security-Policy extension and activate it for the page you'd like to test.

Note - refresh after enabling the extension on a page but before clicking ANDI.

Accessibility Scanning

The tool pa11y-ci is used to scan pages for compliance with a set of accessibility rules. The scan runs as part of our CI setup (see .github/workflows/test.yaml) but it can also be run locally. To run locally, type

docker-compose run pa11y npm run pa11y-ci

The URLs that pa11y-ci will scan are configured in src/.pa11yci. When new views and pages are added, their URLs should also be added to that file.

Security Scanning

The tool OWASP Zap is used for scanning the codebase for compliance with security rules. The scan runs as part of our CI setup (see .github/workflows/test.yaml) but it can also be run locally. To run locally, type

docker-compose run owasp

Images, stylesheets, and JavaScript

We use the U.S. Web Design System (USWDS) for styling our applications.

Static files (images, CSS stylesheets, JavaScripts, etc) are known as "assets".

Assets are stored in registrar/assets during development and served from registrar/public. During deployment, assets are copied from registrar/assets into registrar/public. Any assets which need processing, such as USWDS Sass files, are processed before copying.

Note: Custom images are added to /registrar/assets/img/registrar, keeping them separate from the images copied over by USWDS. However, because the /img/ directory is listed in .gitignore, any files added to /registrar/assets/img/registrar will need to be force added (i.e. git add --force <img-file>) before they can be deployed.

We utilize the uswds-compile tool from USWDS to compile and package USWDS assets.

Making and viewing style changes

When you run docker-compose up the node service in the container will begin to watch for changes in the registrar/assets folder, and will recompile once any changes are made.

Within the registrar/assets folder, the _theme folder contains three files initially generated by uswds-compile:

  1. _uswds-theme-custom-styles contains all the custom styles created for this application
  2. _uswds-theme contains all the custom theme settings (e.g. primary colors, fonts, banner color, etc..)
  3. styles.css a entry point or index for the styles, forwards all of the other style files used in the project (i.e. the USWDS source code, the settings, and all custom stylesheets).

You can also compile the Sass at any time using npx gulp compile. Similarly, you can copy over other static assets (images and javascript files), using npx gulp copyAssets.

CSS class naming conventions

We use the CSS Block Element Modifier (BEM) naming convention for our custom classes. This is in line with how USWDS approaches their CSS class architecture and helps keep our code cohesive and readable.

Upgrading USWDS and other JavaScript packages

  1. Version numbers can be manually controlled in package.json. Edit that, if desired.
  2. Now run docker-compose run node npm update.
  3. Then run docker-compose up to recompile and recopy the assets, or run docker-compose updateUswds if your docker is already up.
  4. Make note of the dotgov changes in uswds-edited.js.
  5. Copy over the newly compiled code from uswds.js into uswds-edited.js.
  6. Put back the dotgov changes you made note of into uswds-edited.js.
  7. Examine the results in the running application (remember to empty your cache!) and commit package.json and package-lock.json if all is well.

Finite State Machines

In an effort to keep our domain logic centralized, we are representing the state of objects in the application using the django-fsm library. See the ADR number 15 for more information on the topic.

Login Time Bug

If you are seeing errors related to openid complaining about issuing a token from the future like this:

ERROR [djangooidc.oidc:243] Issued in the future

it may help to resync your laptop with time.nist.gov:

sudo sntp -sS time.nist.gov

Adding a S3 instance to your sandbox

This can either be done through the CLI, or through the cloud.gov dashboard. Generally, it is better to do it through the dashboard as it handles app binding for you.

To associate a S3 instance to your sandbox, follow these steps:

  1. Navigate to https://dashboard.fr.cloud.gov/login
  2. Select your sandbox from the Applications tab
  3. Click Services on the application nav bar
  4. Add a new service (plus symbol)
  5. Click Marketplace Service
  6. For Space, put in your sandbox initials
  7. On the Select the service dropdown, select s3
  8. Under the dropdown on Select Plan, select basic-sandbox
  9. Under Service Instance enter getgov-s3 for the name and leave the other fields empty

See this resource for information on associating an S3 instance with your sandbox through the CLI.

Testing your S3 instance locally

To test the S3 bucket associated with your sandbox, you will need to add four additional variables to your .env file. These are as follows:

AWS_S3_ACCESS_KEY_ID = "{string value of `access_key_id` in getgov-s3}"
AWS_S3_SECRET_ACCESS_KEY = "{string value of `secret_access_key` in getgov-s3}"
AWS_S3_REGION = "{string value of `region` in getgov-s3}"
AWS_S3_BUCKET_NAME = "{string value of `bucket` in getgov-s3}"

You can view these variables by running the following command:

cf env getgov-{app name}

Then, copy the variables under the section labled s3.

Disable email sending (toggling the disable_email_sending flag)

  1. On the app, navigate to \admin.
  2. Under models, click Waffle flags.
  3. Click the disable_email_sending record. This should exist by default, if not - create one with that name.
  4. (Important) Set the field everyone to Yes. This field overrides all other settings

Request Flow FSM Diagram

The .gov Domain Request & Domain Status Digram visualizes the domain request flow and resulting domain objects.