Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: add pgbouncer #114

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat!: add pgbouncer #114

wants to merge 2 commits into from

Conversation

hrodmn
Copy link
Contributor

@hrodmn hrodmn commented Jan 3, 2025

Add option to set up a PgBouncer server that can efficiently route traffic to the actual database

⚠️ Checklist if your PR is changing anything else than documentation

Merge request description

Add option to set up a pgbouncer server that can manage traffic to the actual database.

Adding a connection pooler like PgBouncer to eoAPI drastically improves performance by preventing the database instance from being overwhelmed by connections from eager clients like titiler-pgstac. AWS offers the RDS Proxy as a managed version of this setup, but it is quite expensive and can double the cost of running an RDS instance. By setting up an EC2 instance running PgBouncer, we can get the same impact for a fraction of the RDS Proxy cost (<$10/mo).

Most of the work to get PgBouncer running was done in MAAP-Project/maap-eoapi#39. This PR builds off of that work and does a more complete job of integrating the PgBouncer server with the rest of the eoAPI applications (not just titiler-pgstac).

I did that by creating a new secret that contains the same values as the original PgstacSecret but with the PgBouncer EC2 instance private IP address substituted for the host key. If the user specifies addPgBouncer: true in the PgStacDatabase construct, an EC2 instance running PgBouncer will be created and the pgstac_secret attribute will be set to the PgBouncer secret instead of the original PgStac secret. This makes it easy to configure client apps to connect to the database via PgBouncer instead of making a direct connection to the database.

The PgBouncer infrastructure is slightly more complicated than I want it to be because we cannot create the PgBouncerSecret until the EC2 instance has been created and assigned an IP address. To get around this, I set up a Lambda function to create the PgBouncer secret with nothing in it, then update it with the correct values after the EC2 instance is running.

Breaking change

The db argument in the TiPgApiLambda, PgStacApiLambda, and TitilerPgstacApiLambda constructs has been replaced withconnectionTarget. This object was/is used to add the Lambdas to the list of acceptable connections and now it can be either an RDS Instance or an EC2 instance (running PgBouncer). I made the breaking change because it would be very easy for an existing user to add PgBouncer to the PgStacDatabase construct then fail to use it when creating the downstream application constructs. By making the breaking change it should force users to at least re-specify the argument name (but since the argument can be either an RDS instance or an EC2 Instance they could easily still mistakenly specify the RDS instance :/).

^ I'm still thinking about the best way to handle that. It feels like those constructs should just accept the whole PgStacDatabase object as an argument rather than the lower-level pieces. Then we could handle that logic in the constructs rather than making the user make unnecessary choices later on. It makes sense to keep it the way it is if we expect users to bring their own db rather than using the PgStacDatabase construct, though.

@hrodmn hrodmn force-pushed the feat/pgbouncer branch 5 times, most recently from 9c69644 to 352bea8 Compare January 3, 2025 17:47
Add option to set up a pgbouncer server that can manage traffic to the
actual database

BREAKING CHANGE: The `db` argument in the TiPgApiLambda, PgStacApiLambda,
and TitilerPgstacApiLambda constructs has been replaced with
`connectionTarget`.
This object is used to add the Lambdas to the list of acceptable
connections and can be either an RDS Instance or an EC2 instance
running pgbouncer.

also install new pgbouncer-secret-updater package in ci
@hrodmn hrodmn changed the title feat: add pgbouncer feat!: add pgbouncer Jan 4, 2025
@hrodmn hrodmn marked this pull request as ready for review January 6, 2025 11:10
@hrodmn hrodmn marked this pull request as draft January 8, 2025 02:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant