Store enriched projects in DynamoDB

[pvhee]

We need to store projects in a NoSQL database like dynamoDB, enriched with:

• organisation data? maybe in a different table
• geo-data. Geocode the main organisation.

In order not to timeout the lambda functions, we might need to use SQS to queue up all projects to be enriched (from the original xls data), process them with a lambda function, and store them enriched in our data storage.

[pvhee]

Getting some insight into timeouts and db throughput tweaking via https://medium.com/@CodingJoe/dealing-with-dynamodb-write-capacity-limits-and-lambda-timeouts-f4e08d9f4b4f#.beltqobit and https://hackernoon.com/top-5-lessons-learned-from-trying-to-build-my-own-serverless-website-d65a168c5e6d#.ylm7i7xa2

This way - we can avoid complicating our architecture with SQS and stick with a DynamoDB. We could probably rate limit writing to dynamoDB via https://github.com/jhurliman/node-rate-limiter

Geo-annotations can happen via streaming to lambda's, following https://aws.amazon.com/blogs/aws/dynamodb-update-triggers-streams-lambda-cross-region-replication-app/

[pvhee]

After playing around with various different ways of dealing with DynamoDB, dynamodb-wrapper seems the most fit to write a lot records without having to have massive write throughput: https://www.npmjs.com/package/dynamodb-wrapper

[pvhee]

Updated plan is to:

• parse projects with related organisations. Store the main organisation at the same level as the project, and store the other organisations as a blob within that project (this is optional)
• enrich each project with geodata -- trial dynamoDB streams with geotagging lambda