poller-lambdas

[twrichards]

This PR introduces the concept of and an abstraction for 'polling' any data source we like and queuing things up for ingestion (via the existing ingestion-lambda).

This abstraction takes care of the infrastructure (see below), build, deployment and boiler plate (obtaining secrets, writing to ingestion queue etc.) so that each 'poller-lambda' implementation (in the new poller-lambdas/src/pollers directory) is focussed purely on the business logic of requesting from the supplier and transforming items with proprietary shape into one or more of our shared IngestorPayloads.

It supports both 'long-polling' and the more traditional fixed-frequency polling (with a frequency in seconds, not constrained by min. 1 min frequency of Cloudwatch/Evenbridge rules). In both cases, it can pass a string output from the invocation to be used as the input for the next (e.g. for passing a pointer id/timestamp etc, but theoretically anything via stringified JSON, albeit transferred in plain text).

It has a useful interactive local run mode, launched by npm run dev in the poller-lambdas directory, to facilitate easy iteration when onboarding new suppliers (see README).

Infrastructure

For every poller (defined in POLLERS_CONFIG in shared/pollers.ts), the following CDK resources are provisioned and wired together...

• Secret's Manager Secret - named according to STAGE and name of the poller, and populated with an arbitrary value ready to be updated with the real value from the supplier (typically holds an API key)
• Lambda - uses the same zip for all the implementations (since they're tiny, and its easier that way) but references the correct handler export
• SQS Queue - this is used to queue further invocations of the lambda, with whatever delay is needed and pass on the output of one invocation as input for the next invocation (the lambda is wired up to process things from the queue)

Each lambda is given permission to read the secret and to write to the ingestion queue and its own queue.

Each lambda has alarms for errors & throttling (both zero-tolerance) plus custom alarms for when the invocation rate is too high or low, which might indicate something has gone haywire.

Thanks to GuCDK, plus a bit of tinkering with the contentDirectory (in cdk/bin/cdk.ts) for the auto-generated riff-raff.yaml the deployment is also abstracted for all pollers.

Any other TODOs are things to be explored in future, not in this PR.

[github-actions]

Deploy build 489 of Editorial Tools::News Wires to CODE

All deployment options

• Deploy build 489 of Editorial Tools::News Wires to CODE
• Deploy parts of build 489 to CODE by previewing it first
• What's on CODE right now?

---

From guardian/actions-riff-raff.

[bryophyta]

is there a particular reason for creating the source queue as part of a different stack?

[twrichards]

I think because FIP have a reference to it, so I think we wanted to decouple, such that we could theoretically tear down the newswires stack without impacting FIP writing to the queue. @andrew-nowak could probably confirm...

[bryophyta]

we discussed off-thread that the reason for using ts-node-dev here is because tsx restarts if the user presses Enter, which doesn't work with the current CLI setup

[twrichards]

yes, swapped out the top-level dependency on tsx so needed to change it in cdk.json too (worth noting GuCDK intialised this command to one using npx which seemed a shame so I can see why it was changed to tsx at one point)

[bryophyta]

💜🙌

[bryophyta]

I've tested locally and read through the code and this looks good to me. Great work! Should make it very easy to get the AP and Reuters pollers set up 👍

[andrew-nowak]

great work! all looks well thought-out and documented. I've left just a couple of non-blocking thoughts and questions