Add Aurora Postgres Connections #425

mojotalantikite · 2021-09-09T19:47:46Z

Summary

This change brings in the configurations that I tested out in #375 and #351 in a clean branch. I wanted serverless to attempt a new deploy via a new stage after all the testing that happened in other branches.

It adds:

Prisma bundled up with webpack in order to generate schemas and establish DB connections
A handler which we can test the postgres connection. It pulls the DB secrets from secrets manager, connects to aurora via Prisma, and executes a test query that pulls basic analytics from the cluster.

Related issues

https://qmacbis.atlassian.net/browse/OY2-10859

People are having issues with newer versions serverless-heaven/serverless-webpack/issues/690

services/app-api/package.json

macrael

Looks great, I'm so excited for this.

macrael · 2021-09-09T22:17:04Z

services/app-api/prisma/schema.prisma

+}
+
+// A model is required for prisma. Dummy model for our testing.
+model TestUser {


Does this table exist in our postgres db? Where did that get created?

It doesn't yet. Prisma refuses to generate a schema without any models and I was just trying to get Prisma client working along with bundling it up into the lambda. So this is just me adding a dummy model to get Prisma to run. I was going to do the migrations work in this epic.

macrael · 2021-09-09T22:18:27Z

services/app-api/webpack.config.js

@@ -0,0 +1,73 @@
+const CopyWebpackPlugin = require('copy-webpack-plugin');


Do we have to use require() here? Everywhere else on the project we're doing:

import * as CopyWebpackPlugin from 'copy-webpack-plugin'

or import CopyWebpackPlugin from 'copy-webpack-plugin, whichever works.

I was actually hitting a bunch of weirdness with webpack when trying to use import statements. Here's the error when you switch to imports:

Syntax Error -------------------------------------------- /Users/mojo/clients/managedcare/managed-care-review/services/app-api/webpack.config.js:1 import CopyWebpackPlugin from 'copy-webpack-plugin'; SyntaxError: Cannot use import statement outside a module

And then when you move package.json to define as a module you get:

Error --------------------------------------------------- Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /Users/mojo/clients/managedcare/managed-care-review/services/app-api/webpack.config.js require() of ES modules is not supported. require() of /Users/mojo/clients/managedcare/managed-care-review/services/app-api/webpack.config.js from /Users/mojo/clients/managedcare/managed-care-review/services/app-api/node_modules/serverless-webpack/lib/validate.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "m odule" which defines all .js files in that package scope as ES modules. Instead rename webpack.config.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /Users/mojo/clients/managedcare/managed-care-review/services/app-api/package.json.

I'm sure there's a fix I just don't know about. I can try to fix that up.

Don't waste too much time on it, this is a config file

I believe you simply can't use es6 import syntax in a file also using module.exports format. Its a node thing - the way this file is written is called (I think) commonjs format - and its different than es6 modules like is used in a our api and web code that we build.

macrael · 2021-09-09T22:22:41Z

src/dev/local/postgres.ts

+    )
+}
+
+export const installPrismaDepsOnce = once(installPrismaDeps)


Nothing wrong with this, but probably unnecessary, I don't think we ever call installAPIDeps twice in the same script. (we do with installWebDeps when you run web and storybook)

Cool, I was just trying to follow the example of how we're doing graphql things, but if you think it's not needed I'm happy to drop using once.

Right, with graphQL we need once b/c it's started by both web and api but this is only run by api

Makes sense, I removed the once wrappers here then.

mojotalantikite · 2021-09-09T20:37:25Z

services/app-api/serverless.yml

+    includeModules: true
+    excludeRegex: 'darwin|debian'
+    packagerOptions:
+      noFrozenLockfile: true


Just wanted to highlight this. Webpack kept throwing errors unless noFrozenLockfile was set to true for us. This might impact reproducibility of our dependencies, so figured we should talk about it.

oh weird, so it's possible for web pack to update our lockfile?

I feel like this is fine. We are still figuring all of this out.

I am curious though what you @mojotalantikite tried to address the issue (roughly). Should we backlog a ticket to revisit this briefly when we finish the migration - I can take that work on if I can get the ctkey stuff working. From what I can tell, we should be able to install with a frozen lockfile (was reading prisma docs and the serverless-webpack docs) so It could be something super minor about our package.json or the webpack conifg being still not quite right.

re: macrae pretty sure any time you run yarn, which webpack is also doing for us to bundle stuff, your lockfile can change randomly

@haworku I tried re-generating the lockfile a couple times as well as trying to see if there were deps that we were using that weren't playing well with webpack. There is a thread here I was reading a couple days ago and it likely is something to do with the dependencies not being in the right spot.

I'll give them another look over to see if I can spot it. At the time I just set this as true out of frustration so I could keep it moving.

This shouldn't block our work here. Just was curious. No need to continue on the pain now just wondering if we should do as a cleanup thing. Yeah could be something that its devDependencies should actually be in dependencies ugh

@haworku I just figured it out. I re-ran yarn add at the point release for all the tools that get packaged up with the nodeExternals() function in webpack and for whatever reason @aws/dynamodb-data-mapper is a dependency that was not in our package.json. I'm not sure where that got dropped or if it just was a dependency that was pulled in via other dependencies and existed in our lock file, but it looks like webpack is packaging it properly now. I dropped the noFrozenLockfiles piece right now and am waiting to see if it will pass CI.

mojotalantikite · 2021-09-09T20:40:31Z

services/app-api/serverless.yml

+      securityGroupIds:
+        - ${self:custom.sgId}
+      subnetIds: ${self:custom.privateSubnets}
+


Wanted to highlight this as well. In order to connect to Aurora, the lambda needs to be in the same VPC. Aurora has another sort of endpoint that doesn't require the lambda to be in the VPC, but Prisma doesn't support it at this point.

services/app-api/package.json

mojotalantikite · 2021-09-10T15:24:05Z

services/app-api/prisma/schema.prisma

+}
+
+// A model is required for prisma. Dummy model for our testing.
+model TestUser {


It doesn't yet. Prisma refuses to generate a schema without any models and I was just trying to get Prisma client working along with bundling it up into the lambda. So this is just me adding a dummy model to get Prisma to run. I was going to do the migrations work in this epic.

mojotalantikite · 2021-09-10T15:30:48Z

services/app-api/webpack.config.js

@@ -0,0 +1,73 @@
+const CopyWebpackPlugin = require('copy-webpack-plugin');


I was actually hitting a bunch of weirdness with webpack when trying to use import statements. Here's the error when you switch to imports:

Syntax Error -------------------------------------------- /Users/mojo/clients/managedcare/managed-care-review/services/app-api/webpack.config.js:1 import CopyWebpackPlugin from 'copy-webpack-plugin'; SyntaxError: Cannot use import statement outside a module

And then when you move package.json to define as a module you get:

Error --------------------------------------------------- Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /Users/mojo/clients/managedcare/managed-care-review/services/app-api/webpack.config.js require() of ES modules is not supported. require() of /Users/mojo/clients/managedcare/managed-care-review/services/app-api/webpack.config.js from /Users/mojo/clients/managedcare/managed-care-review/services/app-api/node_modules/serverless-webpack/lib/validate.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "m odule" which defines all .js files in that package scope as ES modules. Instead rename webpack.config.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /Users/mojo/clients/managedcare/managed-care-review/services/app-api/package.json.

I'm sure there's a fix I just don't know about. I can try to fix that up.

mojotalantikite · 2021-09-10T15:32:43Z

src/dev/local/postgres.ts

+    )
+}
+
+export const installPrismaDepsOnce = once(installPrismaDeps)


Cool, I was just trying to follow the example of how we're doing graphql things, but if you think it's not needed I'm happy to drop using once.

mojotalantikite added 30 commits August 31, 2021 14:43

Try prisma stuff on new branch

7124c9f

Try prisma stuff on new branch

3ab1499

Bump serverless-bundle. See if we break.

48b6cdd

Try npm packager

0a97468

Revert serverless-bundle

7278a34

Npm works on new verison. Not old.

d7957bf

Add esbuild true

9e880c8

Match eslint to bundle versions

cdfa8a1

Be explicit about postgres version

af2acc4

set useUnknownInCatchVariables to false

bfc3be6

Revert engineversion for now

d12d03d

Try with npm

a1f5a72

Add copyFiles back

dd2c345

External the prisma/cli

0b5ad59

Try to use webpack directly

7fb9201

Fix config errors

c00c0e7

Move back to using js webpack config

5bc9999

Remove unused packages

be064e2

Add tsconfig paths for loading

91bb8da

Include performance things from serverless-bundle

73a5abe

Yarn run the prisma

6397e09

Move devdeps to devdeps

98fab66

Downgrade serverless-webpack

cb9d446

People are having issues with newer versions serverless-heaven/serverless-webpack/issues/690

Add the yarn.lock

89a55dd

Remove comment

62c5899

Drop the run. Not needed.

ac1caa8

Use new copywebpack syntax

870d076

Bump back to 5.5.3

b2297bc

Try with noFrozenLockfile

09401db

Merge branch 'main' into mt-connect-aurora

8d7585b

mojotalantikite added 10 commits September 9, 2021 11:57

excludeRegex to all of darwin

a8512b3

Exclude debian. Could be built if native is ubuntu

e967ae3

Add aurora arn to IAM

2974814

Fix arn naming on aurora. Doesn't support getatt

18665e0

Add the schema name to connection string

2b30db1

Fix typo in manual query

7e14991

Move the postgres lambda into the vpc

7a01cb5

Typo fix

bbe7dd3

Merge branch 'mt-connect-aurora' into mt-aurora-handler

3ad7c2e

Cleanup. Add back module

8c92f76

mojotalantikite mentioned this pull request Sep 9, 2021

[test] Aurora Connect #375

Closed

mojotalantikite changed the title ~~Add Aurora Postgres~~ Add Aurora Postgres Connections Sep 9, 2021

haworku reviewed Sep 9, 2021

View reviewed changes

services/app-api/package.json Show resolved Hide resolved

mojotalantikite added 2 commits September 9, 2021 17:19

Fix up some stray dependencies in api

957d599

Get the right eslint-plugin-prettier version

d41eed9

github-actions bot requested a deployment to review-apps September 9, 2021 21:23 In progress

github-actions bot deployed to review-apps September 9, 2021 21:28 View deployment

macrael approved these changes Sep 9, 2021

View reviewed changes

github-actions bot deployed to review-apps September 9, 2021 23:14 View deployment

github-actions bot deployed to review-apps September 9, 2021 23:37 View deployment

mojotalantikite commented Sep 10, 2021

View reviewed changes

haworku approved these changes Sep 10, 2021

View reviewed changes

Remove once wrapper on prisma deps

71c243c

github-actions bot deployed to review-apps September 10, 2021 18:49 View deployment

github-actions bot deployed to review-apps September 10, 2021 19:40 View deployment

Run yarn add again on all nodeExternals

f82cb29

github-actions bot deployed to review-apps September 10, 2021 20:56 View deployment

github-actions bot deployed to review-apps September 10, 2021 21:26 View deployment

mojotalantikite merged commit 697f142 into main Sep 13, 2021

mojotalantikite deleted the mt-aurora-handler branch September 13, 2021 14:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Aurora Postgres Connections #425

Add Aurora Postgres Connections #425

mojotalantikite commented Sep 9, 2021 •

edited

Loading

macrael left a comment

macrael Sep 9, 2021

mojotalantikite Sep 10, 2021

macrael Sep 9, 2021

macrael Sep 9, 2021

mojotalantikite Sep 10, 2021

macrael Sep 10, 2021

haworku Sep 10, 2021

macrael Sep 9, 2021

mojotalantikite Sep 10, 2021

macrael Sep 10, 2021

mojotalantikite Sep 10, 2021

mojotalantikite Sep 9, 2021

macrael Sep 10, 2021

haworku Sep 10, 2021

mojotalantikite Sep 10, 2021

haworku Sep 10, 2021 •

edited

Loading

mojotalantikite Sep 10, 2021

mojotalantikite Sep 9, 2021

mojotalantikite Sep 10, 2021

mojotalantikite Sep 10, 2021

mojotalantikite Sep 10, 2021

		@@ -0,0 +1,73 @@
		const CopyWebpackPlugin = require('copy-webpack-plugin');

Add Aurora Postgres Connections #425

Add Aurora Postgres Connections #425

Conversation

mojotalantikite commented Sep 9, 2021 • edited Loading

Summary

Related issues

macrael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haworku Sep 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mojotalantikite commented Sep 9, 2021 •

edited

Loading

haworku Sep 10, 2021 •

edited

Loading