implement overpack

[jadudm]

At a glance

In order to offer search for an entire domain
as a partner
I want to be able to combine subdomains into a single, overarching search.

Acceptance Criteria

We use DRY behavior-driven development wherever possible.

then...

Preview Give feedback

1. [a thing happens]

Shepherd

• UX shepherd:
• Design shepherd:
• Engineering shepherd:

Background

Right now, we have pack:

sequenceDiagram
  autonumber
  participant Q as Queue
  participant P as Pack
  participant S as S3
  # ---
  Q->>P: get job
  note right of Q: key 
  P->>P: reset timer
  note right of P: timer fires
  note right of P: create sqlitedb
  P->>S: get content
  activate S
  S->>P: JSON
  deactivate S
  note right of P: insert content
  P->>S: store sqlitedb
  P-->>Q: ping search manager

Loading

Essentially, we look for pack messages, and use those to reset a timer. When the timer runs out, we pack a database with the content of the S3 bucket. (This diagram needs to be used to update the docs for pack.)

overpack will work similarly, but it serves a different function.

pack will now enqueue a message for overpack whenever it finishes packing a database.

This will start a timer on overpack. However, overpack will maintain a structure that looks like a reverse dictionary:

alice.va.gov --> va.gov
bob.va.gov --> va.gov
clarice.va.gov --> va.gov
va.gov --> va.gov /* yes, this makes sense */

We will have a second structure that looks like map[domain]timer, much like pack. (A concurrency-safe map.)

So, a ping on alice.va.gov will reset the timer on va.gov.

When the timer for va.gov fires, we'll then create a new database, and proceed to pull the content for each domain in the map. In this case, we would pull any indexed content for the three subdomains, as well as any content indexed on the root domain. (Root domains are essentially have an explicit (or implicit) www subdomain in most cases. They are definitely distinct from named subdomains, and therefore have to be explicitly included.)

After packing, we will ship this like anything else. The serve components need to know that va.gov is available for hosting. It will probably be a large file. There are revisions that are needed to serve regardless, both to handle routing as well as handle larger databases. But, these are known challenges, and not a concern at this time.

sequenceDiagram
  autonumber
  participant Q as Queue
  participant P as Overpack
  participant S as S3
  # ---
  Q->>P: get job
  P->>P: reset timer
  note right of P: timer fires
  note right of P: create sqlitedb
  loop every subdomain
  P->>S: get content for subdomain
  activate S
  S->>P: JSON
  deactivate S
  end
  P->>S: store sqlitedb
  P-->>Q: ping search manager

Loading

Security Considerations

Required per CM-4.

None. All interactions are internal to the app boundaries. We talk to the Postgres DB and S3.

Process checklist

• Has a clear story statement
• Can reasonably be done in a few days (otherwise, split this up!)
• Shepherds have been identified
• UX youexes all the things
• Design designs all the things
• Engineering engineers all the things
• Meets acceptance criteria
• Meets QASP conditions
• Presented in a review
• Includes screenshots or references to artifacts
• Tagged with the sprint where it was finished
• Archived

If there's UI...

• Screen reader - Listen to the experience with a screen reader extension, ensure the information presented in order
• Keyboard navigation - Run through acceptance criteria with keyboard tabs, ensure it works.
• Text scaling - Adjust viewport to 1280 pixels wide and zoom to 200%, ensure everything renders as expected. Document 400% zoom issues with USWDS if appropriate.