📑 Make the crawling process smarter

[jadudm]

At a glance

In order to do less work
as a crawler
I want to be smart.

Acceptance Criteria

We use DRY behavior-driven development wherever possible.

then...

Beta Give feedback

1. [a thing happens]

Shepherd

• UX shepherd:
• Design shepherd:
• Engineering shepherd:

Background

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Unmodified-Since

There's a number of things we could do when we make the HEAD request that would make the crawler smarter. For example, we have the ETag in S3 (when it has been visited before). Therefore, we should be able to grab that, and pass it with a conditional header, and decide if we can just skip the content wholesale.

These kinds of smarts would make the crawling (entree, fetch) much cleaner.

Security Considerations

Required per CM-4.

---

Process checklist

• Has a clear story statement
• Can reasonably be done in a few days (otherwise, split this up!)
• Shepherds have been identified
• UX youexes all the things
• Design designs all the things
• Engineering engineers all the things
• Meets acceptance criteria
• Meets QASP conditions
• Presented in a review
• Includes screenshots or references to artifacts
• Tagged with the sprint where it was finished
• Archived

If there's UI...

• Screen reader - Listen to the experience with a screen reader extension, ensure the information presented in order
• Keyboard navigation - Run through acceptance criteria with keyboard tabs, ensure it works.
• Text scaling - Adjust viewport to 1280 pixels wide and zoom to 200%, ensure everything renders as expected. Document 400% zoom issues with USWDS if appropriate.