Have you ever found yourself scratching your head when your AWS Lambda function works perfectly in one region but fails in another?

Recently, I encountered this exact scenario when moving a function from us-east-1 to eu-west-1.

The culprit? VPC endpoints and Secrets Manager access.

Let's dive into the problem and its solution.

The Problem

Our Lambda function, happily running in us-east-1, suddenly threw this error when deployed to eu-west-1:

An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::ACCOUNT_ID:assumed-role/ROLE_NAME is not authorized to perform: secretsmanager:GetSecretValue on resource: SECRET_ARN because no identity-based policy allows the secretsmanager:GetSecretValue action

The puzzling part? The IAM permissions were identical in both regions.

The Root Cause

After some investigation, i uncovered the core issue: VPC endpoints. In us-east-1, AWS often provides default VPC endpoints for common services, especially in older accounts or those part of organizations with pre-configured settings. However, these default endpoints don't exist in eu-west-1.

Our Lambda function, configured to run in a VPC for enhanced security, couldn't reach Secrets Manager in eu-west-1 without an explicit VPC endpoint.

The Solution

The fix involves creating a VPC endpoint for Secrets Manager in my eu-west-1 VPC. Here's how i did it using AWS CDK:

1. First, i modified my VPC construct to include an option for creating a Secrets Manager endpoint:

export interface VpcConstructProps extends cdk.StackProps {
    // ... other properties ...
    readonly hasSecretsManagerEndpoint?: boolean;
}

const defaultProps: Partial<VpcConstructProps> = {
    // ... other defaults ...
    hasSecretsManagerEndpoint: false,
};

1. Then, in the VPC construct's constructor, i added logic to create the endpoint:

constructor(parent: Construct, name: string, props: VpcConstructProps) {
    // ... existing constructor code ...

    if (props.hasSecretsManagerEndpoint) {
        const secretsManagerEndpoint = vpc.addInterfaceEndpoint("SecretsManagerEndpoint", {
            service: cdk.aws_ec2.InterfaceVpcEndpointAwsService.SECRETS_MANAGER,
        });

        // Suppress cdk-nag warnings
        NagSuppressions.addResourceSuppressions(
            secretsManagerEndpoint,
            [
                {
                    id: 'AwsSolutions-EC23',
                    reason: 'Security Group created by VPC Endpoint allows internal VPC traffic only'
                }
            ],
            true
        );
    }

    // ... rest of constructor code ...
}

1. Finally, when creating my VPC, i enabled the Secrets Manager endpoint:

const vpc = new VpcConstruct(this, 'MyVPC', {
    // ... other props ...
    hasSecretsManagerEndpoint: true,
});

create a vcp endpoint via AWS CLI

#create vcp endpoint via AWS CLI

# Example VPC details

VPC_ID="vpc-12345abcdef"
SUBNET_IDS="subnet-11111aaaaa subnet-22222bbbbb"
SECURITY_GROUP_ID="sg-33333ccccc"
REGION="eu-west-1"

# Create the VPC endpoint
aws ec2 create-vpc-endpoint \
    --vpc-id $VPC_ID \
    --vpc-endpoint-type Interface \
    --service-name com.amazonaws.$REGION.secretsmanager \
    --subnet-ids $SUBNET_IDS \
    --security-group-ids $SECURITY_GROUP_ID \
    --region $REGION


# To verify the endpoint was created:
aws ec2 describe-vpc-endpoints \
    --filters Name=vpc-id,Values=$VPC_ID \
    --region $REGION

The Outcome

After deploying these changes, my Lambda function in eu-west-1 could successfully access Secrets Manager, just like in us-east-1. The error disappeared, and my application worked consistently across regions.

Lessons Learned

1. Regional Differences Matter: Always be aware that AWS services and default configurations can vary between regions.
2. VPC Endpoints are Crucial: When running Lambda in a VPC, consider which AWS services you need to access and ensure appropriate VPC endpoints are in place.
3. Explicit is Better than Implicit: While some regions might have default endpoints, it's better to explicitly define your infrastructure needs in your IaC code.
4. Test in Multiple Regions: Always test your applications in all regions where you plan to deploy.

By understanding and addressing these nuances, we can build more robust, portable, and predictable cloud applications. Happy coding!