Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pluto-message-ingestion: remove externals from ncc config #1180

Merged
merged 2 commits into from
Dec 3, 2024
Merged

pluto-message-ingestion: remove externals from ncc config #1180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
80544cb
pluto-message-ingestion: remove externals from ncc config
andrew-nowak Nov 25, 2024
7ef76fc
include brief testing instructions for the pluto message ingestion la…
andrew-nowak Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 24 additions & 5 deletions pluto-message-ingestion/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,32 @@

An AWS Lambda that provides integration with [Pluto](https://github.com/guardian/pluto) via a Kinesis stream.

Pluto and MAM do not communicate directly as the AWS VPCs are not setup. Instead, they communicate via a Kinesis stream.
Pluto and MAM do not communicate directly as the AWS VPCs are not setup. Instead, they communicate via a Kinesis stream.
Pluto puts JSON messages on this stream and this Lambda picks them up.

The Lambda switches on `message.type` and calls the relevant MAM end-point.

## Message types processed

> [!INFO]
> To test the lambda in CODE, instead of sending messages on the stream
> you can create a test event in the Lambda console.
> Start by creating a message in one of the formats below, then base64-encode
> it, then insert that encoded message into an event that looks like:
> ```json
> {
> "Records": [
> {
> "kinesis": {
> "data": "[your encoded message]"
> }
> }
> ]
> }
> ```
> Your fake Pluto project and commission should now be sent to MAM and stored
> in the `media-atom-maker-CODE-pluto-[projects/commissions]-table`s

### `project-created`
Pluto will send a message to Kinesis when a new Project has been created. MAM will then store a copy of the metadata in Dynamo.

Expand Down Expand Up @@ -49,15 +68,15 @@ Pluto sends a JSON blob that looks like:

### Updating pre-existing projects
If an old Pluto Project is updated, a `project-updated` message still gets put on the Kinesis stream.
As MAM has not seen this project before, it does not know what to update and thus the message is discarded,
As MAM has not seen this project before, it does not know what to update and thus the message is discarded,
logging `attempted to update a project that does not exist`.

This means only projects created **after** the Pluto and MAM changes have gone out can be selected as a destination
for Direct Upload.
This means only projects created **after** the Pluto and MAM changes have gone out can be selected as a destination
for Direct Upload.

### Unknown message types
If a message is put on the Kinesis stream that is not recognised, it will be logged and the lambda will succeed.
This is because Kinesis blocks on failure; succeeding takes the message off the stream so we can continue with
This is because Kinesis blocks on failure; succeeding takes the message off the stream so we can continue with
remaining messages.

See [http://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html](http://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html) for more info.
4 changes: 0 additions & 4 deletions pluto-message-ingestion/ncc.config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
{
"externals": {
"aws-sdk": "aws-sdk",
"/aws-sdk(/.*)/": "aws-sdk$1"
},
"minify": true
}
Loading