🐬 implement serp

[jadudm]

At a glance

In order to serve results
as a developer
I want a dynamically routed backend

Acceptance Criteria

We use DRY behavior-driven development wherever possible.

then...

Beta Give feedback

1. [a thing happens]

Shepherd

• UX shepherd:
• Design shepherd:
• Engineering shepherd:

Background

To handle search requests (both WWW and API), it is important to remember that the SERP pages are essentially an HTML wrapper around the API.

sequenceDiagram
  autonumber
  participant W as WWW
  participant SERP
  participant A as API
  participant 0 as Serve0/1/...
  W->>+SERP: req. page
  SERP->>A: search request
  activate A
  A->>+0: route search
  0->>-A: response
  A->>-SERP: response
  SERP->>-W: response

Loading

1. A request comes to the SERP service from a partner
2. The page fires off the request tact in the initial request
3. The API is a dynamic router; it determines which backend to make the call to
4. The server returns JSON
5. The API returns the JSON to the SERP page
6. The SERP renders the result

If it becomes important, serp can share infrastructure with api, and route directly to the s0/1/... services by doubling up on the queue traffic, and having it track things as well. (It won't do assignments, but it could get updates from the servers to find out where content is... this would save the two-hop.)

As a first implementation, the api service should be OK. We are expecting around 5K PPM peak, and we should be able to do 30-40K PPM with a go-based infrastructure for search results.

Security Considerations

Required per CM-4.

These are WWW and API requests against read-only infrastructure. Inputs are sanitized, and we are following all best practices. The data involved is all PII-free/public data.

---

Process checklist

• Has a clear story statement
• Can reasonably be done in a few days (otherwise, split this up!)
• Shepherds have been identified
• UX youexes all the things
• Design designs all the things
• Engineering engineers all the things
• Meets acceptance criteria
• Meets QASP conditions
• Presented in a review
• Includes screenshots or references to artifacts
• Tagged with the sprint where it was finished
• Archived

If there's UI...

• Screen reader - Listen to the experience with a screen reader extension, ensure the information presented in order
• Keyboard navigation - Run through acceptance criteria with keyboard tabs, ensure it works.
• Text scaling - Adjust viewport to 1280 pixels wide and zoom to 200%, ensure everything renders as expected. Document 400% zoom issues with USWDS if appropriate.

[jadudm]

This is the original design for api/balance:

It is more complex than we need right now.

The API is a dynamic router. From a backend perspective:

sequenceDiagram
  autonumber
  participant P as Pack
  participant Q as Queue
  participant A as API
  participant 0 as Serve0/1/...
  participant S as S3
  0->>Q: space avail
  note left of 0: at startup
  Q-->>A: 
  P->>Q: DB ready
  Q-->>A: job
  note left of A: choose server
  A->>+Q: server choice
  Q-->>-0: DB assignment
  0->>S: fetch
  S-->>0: get db
  0-->>+Q: periodic space remianing notice
  Q-->>-A: space remaining

Loading

1. At startup, the servers announce their space. Should be maximum.
2. The api server builds a map of servers and their space.
3. We finish packing or overpacking a database, enqueueing api.
4. API makes a server choice, based on space available (and possibly traffic).
5. api makes the assignment, enqueueing for s0, s1... (each have a queue)
6. The server (e.g. s0, s1) is notified of the DB it needs to grab
7. Grab the DB from S3
8. "
9. Periodically (and, perhaps after grabbing a database), s0/1/... announce how much space they have.
10. api updates the map with the ground-truth numbers from the servers.

All of this is so that, using the queue, we can go from a completed database, through a dynamic space allocation manager (slash dynamic request router) through to a collection of database servers (sqlite files) that can handle search requests.

[jadudm]

See also #35, which was redundant with this ticket.