TiDB Cloud + AWS Cloudformation Bookshop Demo

January 29, 2023 ยท View on GitHub

Introduction

Welcome to the bookshop demo! This project demonstrates how to use AWS CloudFormation, Secrets Manager, API Gateway, and Lambda Functions to create a fully-functional online bookshop using TiDB Cloud as the database-as-a-service (DBaaS). The Secrets Manager service is used to store database-related details, the API Gateway is used to handle HTTP request routes, and the Lambda Functions are used to handle requests and query data from the TiDB Cloud database.

AWS CloudFormation provides a common language to describe and provision all the infrastructure resources in your environment in a safe, repeatable way. This means that all AWS services used in this demo, including Secrets Manager, API Gateway, and Lambda Functions, will be created using CloudFormation.

Prerequisites

Before you can use this project, you will need the following:

  • An AWS account
  • Access to the AWS CloudFormation, Secrets Manager, API Gateway, Lambda services, S3 and IAM Roles
  • An account with TiDB Cloud

Structure

workflow simplified workflow detailed

Before you begin

Create a TiDB Cloud account and get your free trial cluster(Serverless Tier).

Do remember use aws region us-east-1 to create all your aws resources. If you are not in this region, you should change the region in aws-lambda-cloudformation/src/secretManager.ts and create your code bundle by yourself.

๐Ÿš€ Setting up the demo by Cloudformation template

Note:

We upload the code bundle to S3 bucket, so you can directly use code bundle in us-east-1 region.

The URL is https://tidb-lambda-integration.s3.amazonaws.com/cloudformation-index.zip

If you are not in this region, you should change the region in aws-lambda-cloudformation/src/secretManager.ts and create your code bundle by yourself.

To set up the bookshop demo by Cloudformation, follow these steps:

  1. Download the latest release assets to your local machine.
  2. In the assets, you will find two files: cloudformation_template.yml and cloudformation_template.json. These files contain the AWS CloudFormation template that will be used to create all of the necessary resources for the demo, including the Secrets Manager, API Gateway, and Lambda Functions. You can use either the .yml or .json file, depending on your preference.
  3. In the AWS Management Console, navigate to the CloudFormation service.
  4. Click the "Create Stack" button.
  5. Select the "Upload a template file" option, and choose the CloudFormation template file (either cloudformation_template.yml or cloudformation_template.json) that you want to use.
  6. Follow the prompts to complete the stack creation process. This may take a few minutes.

๐Ÿณ Setting up the demo step-by-step

Note:

All resources created in this section will be charged to your AWS account. Please make sure you have enough credits in your account before proceeding.

The defualut region is us-east-1.

If you are not in this region, you should change the region in aws-lambda-cloudformation/src/secretManager.ts and create your code bundle by yourself.

1. Create a Secrets Manager service

Visit Secrets Manager service in AWS Management Console, and click Store a new secret button.

Input TiDBDatabase, TiDBUser, TiDBPassword, TiDBHost and TiDBPort which are the database-related details of your TiDB Cloud cluster. You can find these details in your TiDB Cloud cluster dashboard. image

Do remember input autogenerated/cloudformation/tidbcloud as the secret name, which will be used in the Lambda function. image

2. Create an IAM Role for Lambda

We need to use Secrets Manager in Lambda function, so we need to create an IAM Role for Lambda to access the Secrets Manager service.

Visit IAM service in AWS Management Console, and click Roles button.

Click Create role button, and select AWS service as the trusted entity, and select Lambda as the use case.

Click Next: Permissions button, and search secretsmanager in the search box, and select SecretsManagerReadWrite policy; Search lambda and select AWSLambdaBasicExecutionRole policy.

Click Next button, and input LambdaSecretsManagerRole as the role name, and click Create role button.

3. Create Lambda functions

We will create 4 Lambda functions in this demo.

3.1 Create a Lambda function to get all books / get a book by id

Visit Lambda service in AWS Management Console, and click Create function button.

Select Author from scratch as the blueprint, and input getBookFunction as the function name, and select Node.js 18x as the runtime.

Select Change default execution role - Use an existing role, and select the LambdaSecretsManagerRole role we created in the previous step.

Click Create function button.

Click Upload a .zip file button, and upload the cloudformation-index.zip file in the latest release assets(create your own code bundle if you are not in us-east-1 region).

Click Save button.

Edit Runtime settings of the Lambda function, and set Handler to getBookHandler.handler. Click Save button.

Choose Configuration tab - General configuration, and set Timeout to 30 sec. Click Save button.

We build the Lambda function in the cloudformation-index.zip file, and you can also build it by yourself. See README for more details.

3.2 Create a Lambda function to post a new book / post initial books

Visit Lambda service in AWS Management Console, and click Create function button.

Select Author from scratch as the blueprint, and input postBookFunction as the function name, and select Node.js 18x as the runtime.

Select Change default execution role - Use an existing role, and select the LambdaSecretsManagerRole role we created in the previous step.

Click Create function button.

Click Upload a .zip file button, and upload the cloudformation-index.zip file in the latest release assets(create your own code bundle if you are not in us-east-1 region).

Click Save button.

Edit Runtime settings of the Lambda function, and set Handler to postBookHandler.handler. Click Save button.

Choose Configuration tab - General configuration, and set Timeout to 30 sec. Click Save button.

3.3 Create a Lambda function to delete a book

Visit Lambda service in AWS Management Console, and click Create function button.

Select Author from scratch as the blueprint, and input deleteBookFunction as the function name, and select Node.js 18x as the runtime.

Select Change default execution role - Use an existing role, and select the LambdaSecretsManagerRole role we created in the previous step.

Click Create function button.

Click Upload a .zip file button, and upload the cloudformation-index.zip file in the latest release assets(create your own code bundle if you are not in us-east-1 region).

Click Save button.

Edit Runtime settings of the Lambda function, and set Handler to deleteBookHandler.handler. Click Save button.

Choose Configuration tab - General configuration, and set Timeout to 30 sec. Click Save button.

3.4 Create a Lambda function to update a book

Visit Lambda service in AWS Management Console, and click Create function button.

Select Author from scratch as the blueprint, and input putBookFunction as the function name, and select Node.js 18x as the runtime.

Select Change default execution role - Use an existing role, and select the LambdaSecretsManagerRole role we created in the previous step.

Click Create function button.

Click Upload a .zip file button, and upload the cloudformation-index.zip file in the latest release assets(create your own code bundle if you are not in us-east-1 region).

Click Save button.

Edit Runtime settings of the Lambda function, and set Handler to putBookHandler.handler. Click Save button.

Choose Configuration tab - General configuration, and set Timeout to 30 sec. Click Save button.

4. Create an API Gateway

Visit API Gateway service in AWS Management Console, and click Create API button.

Select HTTP API as the API type.

Input TiDBCloudApiGatewayV2 as the API name.

Add Integrations for the API Gateway:

  • Click Add integration button, and select Lambda function as the integration type.

  • Select getBookFunction as the Lambda function.

  • Use default settings for other fields.

  • Create other three integrations for postBookFunction, deleteBookFunction and putBookFunction Lambda functions.

Click Next : Configure routes button.

  • Click Add route button, input GET /book as the route name, and select getBookFunction as the integration.

  • Click Add route button, input GET /book/{id} as the route name, and select getBookFunction as the integration.

  • Click Add route button, input POST /book as the route name, and select postBookFunction as the integration.

  • Click Add route button, input POST /book/init as the route name, and select postBookFunction as the integration.

  • Click Add route button, input DELETE /book/{id} as the route name, and select deleteBookFunction as the integration.

  • Click Add route button, input PUT /book/{id} as the route name, and select putBookFunction as the integration.

Click Next : Define stages button.

Use the default settings.

Click Next : Review button.

Click Create button.

Using the demo

Once the stack has been created, you can use the demo as follows:

Visit API Gateway service in AWS Management Console, and click TiDBCloudApiGatewayV2 API.

Check the Invoke URL in the Overview page, and copy it. This is the API endpoint.

Use API test tools like Postman to test the API.

We will use cURL in the following examples. You can also use Postman or other tools.

Init mock books

curl -X POST -H "Content-Type: application/json" -d '{"count":100}' https://<your-api-endpoint>/book/init

Get all books

curl https://<your-api-endpoint>/book

Get a book by id

curl https://<your-api-endpoint>/book/1

Update a book by id

curl -X PUT -H "Content-Type: application/json" -d '{ "title": "Book Title(updated)" }' https://<your-api-endpoint>/book/1

Post a new book

curl -X POST -H "Content-Type: application/json" -d '{ "title": "Book Title", "type": "Test", "publishAt": "2022-12-15T21:01:49.000Z", "stock": 123, "price": 12.34, "authors": "Test Test" }' https://<your-api-endpoint>/book

Delete a book by id

curl -X DELETE https://<your-api-endpoint>/book/1

Cleaning up

CloudFormation stack

  • Delete the stack in CloudFormation service in AWS Management Console.

  • Delete the S3 bucket in S3 service in AWS Management Console.

Step-by-step guide

  • Delete the TiDBCloudApiGatewayV2 API in API Gateway service in AWS Management Console.

  • Delete the getBookFunction, postBookFunction, deleteBookFunction and putBookFunction Lambda functions in Lambda service in AWS Management Console.

  • Delete the LambdaSecretsManagerRole role in IAM service in AWS Management Console.

  • Delete the TiDBCloudSecretsManager secret in Secrets Manager service in AWS Management Console.

aws-lambda-cloudformation Code Example

This repository contains an AWS Lambda function that uses TiDB Cloud as the database, Sequelize as the ORM, Fastify as the API framework, and AWS Secrets Manager SDK to retrieve connection configurations.

Technologies

  • TiDB Cloud: A cloud-native MySQL-compatible HTAP database.
  • Sequelize: A promise-based Node.js ORM for working with relational databases such as MySQL, MariaDB, and PostgreSQL.
  • Fastify: A web framework for building fast and scalable APIs with Node.js.
  • AWS Secrets Manager SDK: A library for accessing and managing secrets in AWS Secrets Manager.

Installation

  1. Install Node.js 18.x and yarn on your machine.
  2. Clone this repository and navigate to the project directory /aws-lambda-cloudformation.
  3. Run yarn install to install the dependencies.

Usage

  1. Create a TiDB Cloud cluster and retrieve the connection details.
  2. Store the connection details(TiDBDatabase, TiDBUser, TiDBPassword, TiDBHost and TiDBPort which are the database-related details of your TiDB Cloud cluster) as a secret named autogenerated/cloudformation/tidbcloud in AWS Secrets Manager.
  3. Run yarn start to start the server.

Note: AWS Secrets Manager SDK will automatically retrieve the secret value from AWS Secrets Manager and use it to connect to TiDB Cloud. You need to have the permission to access the secret. You need to configure the AWS credentials on your machine to access AWS Secrets Manager. See AWS Secrets Manager SDK for more details.

Build

Run yarn build to build the project.

ESBuild will bundle the project into a single file dist/index.js.

It will also zip the file into dist/index.zip which is the deployment package for AWS Lambda.

Known Issues

There are no known issues at this time.

Contribution Guidelines

If you would like to contribute to this project, please follow these guidelines:

  1. Fork the repository.
  2. Create a new branch for your changes.
  3. Make your changes and commit them to your branch.
  4. Open a pull request to merge your changes into the master branch.

Acknowledgments

  • TiDB Cloud
  • Sequelize: A promise-based Node.js ORM for working with relational databases.
  • Fastify: A web framework for building fast and scalable APIs with Node.js.
  • AWS Secrets Manager SDK: A library for accessing and managing secrets in AWS Secrets Manager.