Engaging Crowds: switch to Datasette for subject browse & search

[eatyourgreens]

Switch the subject indexing tool over to Datasette.

This PR lets you select a subject set and subject to classify. #2148 is needed in order to load the indexed subjects into the classifier.

Test URL:
http://local.zooniverse.org:3000/projects/bogden/scarlets-and-blues?env=production&demo=true
The Minutes workflow for Scarlets & Blues has subject sets which are indexed in Datasette.

Package:
app-project

Review Checklist

General

• Are the tests passing locally and on Travis?
• Is the documentation up to date?

Components

• Has a storybook story been created or updated?
• Is the component accessible?
  • Can it be used with a screen reader? BBC guide to testing with VoiceOver
  • Can it be used from the keyboard? WebAIM guide to keyboard testing
  • Is it passing accessibility checks in the storybook?

Apps

• Does it work in all major browsers: Firefox, Chrome, Edge, Safari?
• Does it work on mobile?
• Can you yarn panic && yarn bootstrap or docker-compose up --build and app works as expected?

Publishing

• Is the changelog updated?
• Are the dependencies updated for apps and libraries that are using the newly published library?

Post-merging

• Did the app deploy to https://frontend.preview.zooniverse.org/projects/:project-name/:owner or https://frontend.preview.zooniverse.org/about?
• Is the new feature working or bug now fixed?
  • Is there a Talk or blog post written to announce the new feature(s)?
• Is the design working across browsers (Firefox, Chrome, Edge, Safari) and mobile?
  • Is this approved by our designer?
• Is this ready for production deployment?

[shaunanoordin]

Awright, code read looks good. I only have two comments (see above), so I'll start with the functionality tests now. Or after dinner. We'll see!

[shaunanoordin]

PR Review

This PR connects the "Subject Set selection" logic to an external Datasette service. This external service essentially acts as a query-able index for information on subject sets, e.g. "yo pls list all Subjects in this set in its published page order"

• New external dependency: https://subject-set-search-api.zooniverse.org/subjects (Datasette service which lists subjects/subject sets)
• Hardcoded values (e.g. "WF 16106 has been indexed") have been removed so logic is more correct. (i.e. pls check with the external Datasette service to see if the subject set has been indexed there)
• ⚠️ Important note: code assumes production environment only, simply because there's no 'staging alternative' to subject-set-search-api.zooniverse.org
• Note: the ping to the external service is done on Next.js, not client-side.

Testing

Tested on localhost, using Chrome + macOS.

URLs:

• Main project: Scarlets and Blues, "Minutes" workflow
• Baseline: Engaging the Crowds, any workflow.

Testing:

• Go to the /classify page
• Choose the workflow, choose any Subject Set.
• For the main project, this should load a list of all Subjects for the Subject Set. ("Select Subject" functionality)
  • NOTE: actually clicking on the specific Subject won't actually load it. This is a separate PR.
• For the baseline, no differences in the functionality should be noted.

Testing looks good 👍 After adding some debug commands into the code, I can confirm that the external Datasette service is being pinged correctly by Next.js.

❓ Advanced testing:

• I tried to figure out what'd happen if hasIndexedSubjects() couldn't reach the external service (e.g. the service is 500-ing).
• In theory, this should bork the main project (no indexing service = this feature shouldn't be able to work, so a functionality crash should be expected) but have no effect on the baseline project (the hasIndexedSubjects just times out and all it costs is a bit of waiting time).
• Unfortunately, I haven't been able to devise a way to test this properly. (Substituting the URL with a fake one didn't quite pan out the way I expected.)

Main goal of this advanced test is to ensure a failure in the external dependency should not negatively affect projects that don't depend on it.

Probably nothing to worry about right now, but worth noting in case it crops up in the future.

Status

LGTM, giving this a 👍

[eatyourgreens]

If the promises in Promise.allSettled(subject_sets.map(subjectSet => hasIndexedSubjects(subjectSet))) all reject, then I think subjectSet.isIndexed will be false by default for each set. At least, that's my reading of the tests, which set isIndexed to false by default. So I think the default behaviour, if the indexing service is down, would be to turn off subject selection in the browser. After selecting a subject set, you'd be taken straight to that set in the classifier.

On the other hand, if Promise.allSettled errors, then the error would be caught and logged, and an empty array of subject sets would be returned. That would break selection for a grouped workflow, because you must choose a set before you can classify in that case.