AWS Lamba & RDS Proxy

Probably like many others, I tried to avoid relational databases like Postgres, MySQL and MariaDB with Lambda as I've read many times that connection based services aren't a good fit for the potentially rapid scaling of Lambda. So ever since I am doing serverless, I try to use DynamoDB or ElastiCache Redis/Valkey as much as possible.

At some point I really needed to use a relational database and I started playing with RDS Aurora. I created an instance, connected from Lambda and it worked just fine. However when I generated a bit more load it soon started locking up, all connections were in use and new ones couldn't be created. It would take a while for the database to become available again. The warning for combining Lambda with connection based services are not there without reasons.

RDS Proxy

Then, I remembered RDS Proxy and RDS Data API to exist and now I understood why. While Data API is a great service, it does come with some limitations and I decided to go with RDS Proxy. I also liked the aspect of being able to use generic libraries for connecting and querying data.

CloudFormation

Getting everything to work with CloudFormation and using best practices took me a while as a different setting could completely lock up the creation of a database. I also ran into issues where the RDS Proxy was created and had no errors in the console but it turned out that, using CLI tools, the network config was broken which was the reason I couldn't connect. To save you from this pain, I created a demo that sets up Lambda with RDS Aurora & Proxy making use of security best practices like a managed password and IAM authentication.

Demo

The demo is using the open source fork of Serverless Framework (a drop-in replacement for Serverless Framework v3!) with Typescript that will give you the following after deploying:

Networking

A basic setup with a VPC, 2 private and 2 public subnets in different availability zones for high availability (thanks to my colleague Martijn).

RDS Aurora Postgres

An RDS Aurora Postgres cluster with Postgres 16.6 with 2 serverless instances for Multi-AZ support. Also enabled are encrypted storage, backups, IAM authentication and serverless scaling configuration.

Database:
  Type: AWS::RDS::DBCluster
  Properties:
    ManageMasterUserPassword: True
    Engine: aurora-postgresql
    EngineVersion: 16.6
    StorageEncrypted: True
    EnableIAMDatabaseAuthentication: True
    BackupRetentionPeriod: 7
    ServerlessV2ScalingConfiguration:
      MinCapacity: 0
      MaxCapacity: 1

RDS Proxy

An RDS Proxy instance with read/write and read only endpoints making use of IAM authentication.

DatabaseProxy:
  Type: AWS::RDS::DBProxy
  Properties:
    Auth:
      - AuthScheme: SECRETS
    ClientPasswordAuthType: POSTGRES_SCRAM_SHA_256
    IAMAuth: REQUIRED
    SecretArn:
      Fn::GetAtt: [ Database, MasterUserSecret.SecretArn ]
    EngineFamily: POSTGRESQL
    RequireTLS: True

Lambda

Finally a Lambda function, security group and role which can connect to the RDS Proxy using IAM authentication. Now we can just use pg with the rds-signer package to work with Postgres the way you are used to!

const db = await getClient();
const result = await db.query('SELECT version()');

console.log(result.rows[0].version);

const db = await getClient();
const result = await db.query(
  'SELECT id, title, body FROM articles WHERE type = $1',
  ['database'],
);

for (const article of result.rows) {
  console.log(article);
}

Certificate

Even though AWS points you towards a certificate for RDS per region like https://truststore.pki.rds.amazonaws.com/eu-west-1/eu-west-1-bundle.pem this does not apply to RDS Proxy for which you will need to use https://www.amazontrust.com/repository/AmazonRootCA1.pem.

Warnings

Obviously this is just a stripped down demo with the basics and you need to modify the template to your own needs. Even though this is mostly a serverless setup, you will be charged for RDS Aurora and RDS Proxy per hour. Because RDS Proxy will stay connected, RDS Aurora will not be able to pause / scale back down to 0:

If your Aurora cluster has an associated proxy through Amazon RDS Proxy, the proxy maintains an open connection to each DB instance in the cluster and the instance doesn’t automatically pause.

Warning: By deploying this demo, you acknowledge that AWS services may incur costs. I, the author, am not responsible for any charges or usage fees associated with your AWS account. Deploy at your own discretion.

Database Management in a VPC

It seems many people are still struggling using database tools because their RDS cluster is running inside a VPC. I recommend you to read one of my previous articles to see the killer tool that solved this problem for me: Port7777.

Closing

After migrating to RDS Proxy, I did not run into performance or scalability issues anymore.

To close of, again the link to the demo: https://github.com/slootjes/aws-lambda-rds-proxy. Let me know if you think I've missed something crucial. In the next article I will explain how to use S3 features with RDS Aurora and update the demo. Have fun!