Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tidb cloud: AWS Lambda integration #13233

Merged
merged 33 commits into from
Jan 21, 2025
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
def7300
add dev guide aws-lambda-integration
shczhen Apr 13, 2023
ae39dd4
fix lint error
shczhen Apr 13, 2023
ca4b37c
Update dev-guide-aws-lambda-integration.md
shczhen Apr 13, 2023
0ffc8d4
Apply suggestions from code review
shczhen Apr 20, 2023
24cbc6b
Update dev-guide-aws-lambda-integration.md
shczhen Apr 20, 2023
46b2699
follow suggestions
shczhen Apr 20, 2023
9e4a801
Update TOC.md
shczhen Apr 20, 2023
850cdbc
Update dev-guide-aws-lambda-integration.md
shczhen Apr 23, 2023
c7fcb22
Apply suggestions from code review
shczhen Apr 25, 2023
dad8bb4
fix link
shczhen Apr 25, 2023
6bf2302
Update dev-guide-aws-lambda-integration.md
shczhen Apr 25, 2023
6819b0a
refine language; adjust format; unify style
ran-huang Apr 26, 2023
1271896
Merge remote-tracking branch 'upstream/release-6.5' into dev-guide-aw…
hfxsd Jan 20, 2025
633c423
polished contents
hfxsd Jan 20, 2025
854ebd4
change to cloud toc
hfxsd Jan 20, 2025
e55e652
Update TOC.md
hfxsd Jan 20, 2025
cb36c07
Update integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 20, 2025
75ef30f
Update integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 20, 2025
101eab9
Update integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 20, 2025
0d6cab9
use details for some contents
hfxsd Jan 21, 2025
ca4fd87
Update integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 21, 2025
45995c2
fix typos
hfxsd Jan 21, 2025
7f077d1
Update integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 21, 2025
b2a0d6d
Update integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 21, 2025
8556bf2
Apply suggestions from code review
hfxsd Jan 21, 2025
992307b
update screenshots
hfxsd Jan 21, 2025
93349af
Update tidb-cloud/integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 21, 2025
d91c609
update screenshots
hfxsd Jan 21, 2025
0b38952
Update tidb-cloud/integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 21, 2025
249ac47
Update tidb-cloud/integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 21, 2025
d711af7
Apply suggestions from code review
hfxsd Jan 21, 2025
bb6bd43
Apply suggestions from code review
hfxsd Jan 21, 2025
e0c5243
Update tidb-cloud/integrate-tidbcloud-with-aws-lambda.md
hfxsd Jan 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions TOC-tidb-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@
- Integrations
- [Airbyte](/tidb-cloud/integrate-tidbcloud-with-airbyte.md)
- [Amazon AppFlow](/develop/dev-guide-aws-appflow-integration.md)
- [AWS Lambda](/tidb-cloud/integrate-tidbcloud-with-aws-lambda.md)
- [Cloudflare](/tidb-cloud/integrate-tidbcloud-with-cloudflare.md)
- [Datadog](/tidb-cloud/monitor-datadog-integration.md)
- [dbt](/tidb-cloud/integrate-tidbcloud-with-dbt.md)
Expand Down
Binary file added media/develop/aws-lambda-cf-create-stack.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added media/develop/aws-lambda-cf-stack-config.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added media/develop/aws-lambda-structure-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
202 changes: 202 additions & 0 deletions tidb-cloud/integrate-tidbcloud-with-aws-lambda.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
---
title: Integrate TiDB Cloud Serverless with Amazon Lambda Using AWS CloudFormation
summary: Introduce how to integrate TiDB Cloud Serverless with Amazon Lambda and CloudFormation step by step.
---

# Integrate TiDB Cloud Serverless with Amazon Lambda Using AWS CloudFormation

This document provides a step-by-step guide on how to use [AWS CloudFormation](https://aws.amazon.com/cloudformation/) to integrate [TiDB Cloud Serverless](https://www.pingcap.com/tidb-cloud/), a cloud-native distributed SQL database, with [AWS Lambda](https://aws.amazon.com/lambda/), a serverless and event-driven compute service. By integrating TiDB Cloud Serverless with Amazon Lambda, you can leverage the scalability and cost-efficiency of microservices through TiDB Cloud Serverless and AWS Lambda. AWS CloudFormation automates the creation and management of AWS resources, including Lambda functions, API Gateway, and Secrets Manager.

## Solution overview

In this guide, you will create a fully functional online bookshop with the following components:

- AWS Lambda Function: handles requests and queries data from a TiDB Cloud Serverless cluster using Sequelize ORM and Fastify API framework.
- AWS Secrets Manager SDK: retrieves and manages connection configurations for the TiDB Cloud Serverless cluster.
- AWS API Gateway: handles HTTP request routes.
- TiDB Cloud Serverless: a cloud-native distributed SQL database.

AWS CloudFormation is used to create the necessary resources for the project, including the Secrets Manager, API Gateway, and Lambda Functions.

The structure of the bookshop project is as follows:

![AWS Lambda structure overview](/media/develop/aws-lambda-structure-overview.png)

## Prerequisites

Before getting started, ensure that you have the following:

- An AWS account with access to the following AWS services:
- [AWS CloudFormation](https://aws.amazon.com/cloudformation/)
- [Secrets Manager](https://aws.amazon.com/secrets-manager/)
- [API Gateway](https://aws.amazon.com/api-gateway/)
- [Lambda services](https://aws.amazon.com/lambda/)
- [S3](https://aws.amazon.com/s3/)
- [IAM Roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)
- A [TiDB Cloud](https://tidbcloud.com) account and a TiDB Cloud Serverless cluster. Get the connection information for your TiDB Cloud Serverless cluster:

![TiDB Cloud connection information](/media/develop/aws-lambda-tidbcloud-connection-info.png)
hfxsd marked this conversation as resolved.
Show resolved Hide resolved

- API test tools such as [Postman](https://www.postman.com/) and [cURL](https://curl.se/). Most examples in this document use cURL. For Windows users, Postman is recommended.
- Download the [latest release assets](https://github.com/pingcap/TiDB-Lambda-integration/releases/latest) of the project to your local machine, which includes `cloudformation_template.yml` and `cloudformation_template.json` files.

> **Note:**
>
> - When you create the AWS resources, it is recommended to use `us-east-1` as your cluster region. This is because the Lambda function code in this demo hardcodes the region as `us-east-1`, and the code bundle is stored in the `us-east-1` region.
> - If you use a different region, you need to follow the following instructions to modify the Lambda function code, rebuild it, and upload the code bundle to your own S3 bucket.

<details>
<summary>Modify and rebuild the Lambda function code if you use a region other than <code>us-east-1</code></summary>

If you use `us-east-1` as your cluster region, skip this section and go to [Step 1: Set up the project using AWS CloudFormation](#step-1-set-up-the-project-using-aws-cloudformation).
hfxsd marked this conversation as resolved.
Show resolved Hide resolved

If you use a different AWS region other than `us-east-1` to create the AWS resources, you need to modify the Lambda function code, rebuild it, and upload the code bundle to your own S3 bucket.

To avoid local development environment issues, it is recommended that you use a cloud-native development environment, such as [Gitpod](https://www.gitpod.io/).

To rebuild and upload the code bundle to your own S3 bucket, do the following:

1. Initialize the development environment.

- Open the [Gitpod](https://gitpod.io/#/https://github.com/pingcap/TiDB-Lambda-integration) workspace and log in with your GitHub account.

2. Modify the Lambda function code.

1. Open the `aws-lambda-cloudformation/src/secretManager.ts` file in the left sidebar.
2. Locate the line 22 and then modify the `region` variable to match your own region.

3. Rebuild the code bundle.

1. Install the dependencies.

1. Open a terminal in Gitpod.
2. Enter the working directory:

```shell
cd aws-lambda-cloudformation
```

3. Install the dependencies:

```shell
yarn
```

2. Rebuild the code bundle.

1. Build the code bundle.

```shell
yarn build
```

2. Check the `aws-lambda-cloudformation/dist/index.zip` file.
3. Right-click the `index.zip` file and select **Download**.

4. Upload the rebuilt code bundle to your own S3 bucket.

1. Visit the [S3 service](https://console.aws.amazon.com/s3) in the AWS Management Console.
2. Create a new bucket in your selected region.
3. Upload the `index.zip` file to the bucket.
4. Note down the S3 bucket name and region for later use.

</details>

## Step 1. Set up the bookshop project using AWS CloudFormation

To set up the bookshop project using AWS CloudFormation, do the following:

1. Navigate to the AWS Management Console and access the [AWS CloudFormation service](https://console.aws.amazon.com/cloudformation).
2. Click **Create Stack** > **With new resources (standard)**.
3. On the **Create Stack** page, complete the stack creation process.

1. In the **Prerequisite** area, select **Choose an existing template**.
2. In the **Specify template** area, select **Upload a template file**, click **Choose file** to upload the template file (either YAML or JSON), and click **Next**.

If you do not have the file yet, download it from [GitHub](https://github.com/pingcap/TiDB-Lambda-integration/releases/latest). The file contains the AWS CloudFormation template that creates the necessary resources for the project.

![Create a stack](/media/develop/aws-lambda-cf-create-stack.png)

3. Specify stack details.

- If you use `us-east-1` as your cluster region, fill in the fields as in the following screenshot:

![Specify AWS Lambda stack details](/media/develop/aws-lambda-cf-stack-config.png)

- **Stack name**: enter the stack name.
- **S3Bucket**: enter the S3 bucket where you store the zip file.
- **S3Key**: enter the S3 key.
- **TiDBDatabase**: enter the TiDB Cloud cluster name.
- **TiDBHost**: enter the host URL for TiDB Cloud database access. Enter `localhost`.
- **TiDBPassword**: enter the password for TiDB Cloud database access.
- **TiDBPort**: enter the port for TiDB Cloud database access.
- **TiDBUser**: enter the user name for TiDB Cloud database access.

- If you use a different AWS region other than `us-east-1`, follow these steps:

1. Refer to [Modify and rebuild the Lambda function code if you use a region other than `us-east-1`](#prerequisites) to modify the Lambda function code, rebuild it, and upload the code bundle to your own S3 bucket.
2. In the stack details fields, specify the S3 bucket name and region in the `S3Bucket` and `S3Key` parameters according to your own configuration.
3. Fill in other fields as in the preceding screenshot.

hfxsd marked this conversation as resolved.
Show resolved Hide resolved
4. Configure stack options. You can use the default configurations.

![Configure stack options](/media/develop/aws-lambda-cf-stack-config-option.png)

5. Review and create the stack.

![Review and create the stack](/media/develop/aws-lambda-cf-stack-config-review.png)

## Step 2. Use the bookshop project

After the stack has been created, you can use the project as follows:

1. Visit the [API Gateway service](https://console.aws.amazon.com/apigateway) in the AWS Management Console, click the `TiDBCloudApiGatewayV2` API, and then click **API: TiDBCloudApiGatewayV2** in the left pane.

2. Copy the `Invoke URL` from the **Overview** page. This URL serves as the API endpoint.

![API Gateway Invoke URL](/media/develop/aws-lambda-get-apigateway-invoke-url.png)

3. Use API test tools such as Postman and cURL to test the API:

- Init mock books:

```shell
curl -X POST -H "Content-Type: application/json" -d '{"count":100}' https://<your-api-endpoint>/book/init
```

- Get all books:

```shell
curl https://<your-api-endpoint>/book
```

- Get a book by the book ID:

```shell
curl https://<your-api-endpoint>/book/<book-id>
```

- Create a book:

```shell
curl -X POST -H "Content-Type: application/json" -d '{ "title": "Book Title", "type": "Test", "publishAt": "2022-12-15T21:01:49.000Z", "stock": 123, "price": 12.34, "authors": "Test Test" }' https:// <your-api-endpoint>/book
```

- Update a book:

```shell
curl -X PUT -H "Content-Type: application/json" -d '{ "title": "Book Title(updated)" }' https://<your-api-endpoint>/book/<book-id>
```

- Delete a book:

```shell
curl -X DELETE https://<your-api-endpoint>/book/<book-id>
```

## Step 3. Clean up resources

To avoid unnecessary charges, clean up all resources that have been created.

1. Access the [AWS Management Console](https://console.aws.amazon.com/cloudformation).
2. Delete the AWS CloudFormation stack that you created.
Loading