CFN AWS SOCI Index Builder on AWS

Deployment Guide

June 2025
AWS Nucleo
Container Runtime

Refer to the GitHub repository to view source files, report bugs, submit feature ideas, and post feedback about this Solution. To comment on the documentation, refer to Feedback.

This Cloud Formation Template was created by Amazon Web Services (AWS). This template helps customers deploy SOCI Index builder stack on AWS according to AWS best practices.

Overview

This guide covers information you need to deploy the Seekable OCI (SOCI) Index Builder to the AWS Cloud.

Costs and licenses

There is no cost to use this template, but you will be billed for any AWS services or resources that this template deploys. For more information, refer to the AWS Quick Starts General Information Guide.

Architecture

Deploying this template with default parameters builds the following SOCI Index Builder environment in the AWS Cloud.

Figure 1. Solution architecture for SOCI Index Builder on AWS

As shown in Figure 1, this solution sets up the following:

An Amazon Elastic Container Registry (Amazon ECR) to store container images and SOCI artifacts.
An AWS CloudFormation stack that contains the following resources:
- An Amazon EventBridge rule to detect Amazon ECR image action events and invoke the filtering AWS Lambda function.
- Two AWS Lambda functions to do the following:
  - Filter Amazon ECR image action events that match the filters provided in the CloudFormation stack parameters.
  - Generate SOCI indices for matching images and push indices back to image repositories in the Amazon ECR registry.
- AWS Identity and Access Management (IAM) to provide the following permissions:
  - EventBridge to invoke the filtering Lambda function.
  - Filtering Lambda function to create Amazon CloudWatch logs and to invoke the SOCI index generator Lambda function.
  - SOCI index generator Lambda function to create CloudWatch logs and access the Amazon ECR registry.
CloudWatch to collect logs from both Lambda functions for troubleshooting.

Predeployment Steps

Prerequisites

Ensure you have the following tools installed:

Git - To clone the repository
Python/pip - Required for taskcat installation
Docker - For packaging Lambda functions
taskcat - Install using: pip install taskcat

# Clone the repository
git clone https://github.com/awslabs/cfn-ecr-aws-soci-index-builder.git
cd cfn-ecr-aws-soci-index-builder

# Verify Docker is running
docker info

Taskcat Configuration

We will use taskcat tool to package and upload the code artifacts. Taskcat is a tool that will build and bundle the CFN template into an S3 Bucket which users can then use to spin up their cloudformation stack.

The configuration for the taskcat tool is present in the .taskcat.yml in the root of the project directory.

Example .taskcat.yml configuration

project:
  name: cfn-ecr-aws-soci-index-builder
  owner: [email protected]
  s3_regional_buckets: true
  lambda_source_path: functions/source
  lambda_zip_path: functions/packages
  parameters:
    QSS3BucketName: $[taskcat_autobucket]
    QSS3KeyPrefix: "cfn-ecr-aws-soci-index-builder/"
tests:
  can-deploy:
    parameters:
      SociRepositoryImageTagFilters: "*:*"
      SociIndexVersion: "V2"
    regions:
      - us-east-1      # US East (N. Virginia)
      - eu-west-1      # Europe (Ireland)
      - us-west-1      # US West (N. California)
      # Add or remove regions as needed
    template: templates/SociIndexBuilder.yml

The .taskcat.yml file must be updated with the correct regions where you want to deploy your stack. This is crucial as the template will be built specifically for those regions.

For AWS GovCloud deployment, update the regions section:

    regions:
      - us-gov-west-1  # AWS GovCloud (US-West)
      - us-gov-east-1  # AWS GovCloud (US-East)

For China Region deployment, update the regions section:

    regions:
      - cn-north-1     # China (Beijing)
      - cn-northwest-1 # China (Ningxia)

Deployment steps

There are two ways to deploy this stack

1. Automatic Deployment

You can automatically deploy the stack directly from the CLI using taskcat. This method requires updating the .taskcat.yml file with your specific requirements, particularly the SociIndexVersion parameter:

Example .taskcat.yml configuration for automatic deployment

project:
  name: cfn-ecr-aws-soci-index-builder
  owner: [email protected]
  s3_regional_buckets: true
  lambda_source_path: functions/source
  lambda_zip_path: functions/packages
  parameters:
    QSS3BucketName: $[taskcat_autobucket]
    QSS3KeyPrefix: "cfn-ecr-aws-soci-index-builder/"
tests:
  can-deploy:
    stack_name: "Soci-index-builder"
    # stack_name_prefix: "PrefixStackname"
    # stack_name_suffix: "SufficStackName"
    parameters:
      SociRepositoryImageTagFilters: "*:*"
      SociIndexVersion: "V2"
    regions:
      - us-east-1
    template: templates/SociIndexBuilder.yml

To deploy the stack, run:

taskcat test run -n

The -n flag prevents the stack from being deleted after deployment, which is the default behavior of test run.

You can customize the stack name using one of these options:

stack_name: Set a specific name
stack_name_prefix: Add a prefix to the name
stack_name_suffix: Add a suffix to the name

While this automatic deployment provides a convenient one-command deployment, it has some limitations:

The SociIndexVersion is set at deployment time and cannot be changed without redeploying the stack
The automatic deployment method sets a fixed SociIndexVersion in the taskcat configuration, using the upload command and console deployment approach allows you to specify different SociIndexVersion values each time you create a new stack from the pre-uploaded template

2. Manual Deployment

If you used the automatic deployment then you can skip these steps and directly go check the cloudformation stack has deployed

After configuring the taskcat.yml file, run taskcat upload from the root directory of the repository.

The upload command will:

Read the configuration from .taskcat.yml file
Package the lambda functions using the provided Dockerfile (this is why Docker is required)
Create S3 buckets in all specified regions with the naming pattern tcat-<hash>-<region>
Upload the CloudFormation template and Lambda packages to these buckets with the prefix cfn-ecr-aws-soci-index-builder/
Make the CloudFormation template available at the URL pattern https://<bucket-name>.s3.<region>.amazonaws.com/cfn-ecr-aws-soci-index-builder/templates/SociIndexBuilder.yml

Before proceeding with the actual deployment, you can run taskcat upload --dry-run to safely preview the resources that will be provisioned. This command simulates the upload process without making any actual changes, helping you verify your configuration.

In the above example you’re deploying to the us-east-2 region, the S3 bucket name created is: tcat-318317e4bf395d3fa89e1d675d64f43e-us-east-2

After the upload completes:

Verify your deployment - Find your S3 bucket (starts with tcat-) in the AWS Console
Locate the template at cfn-ecr-aws-soci-index-builder/templates/SociIndexBuilder.yml
Copy the Object URL:

You can also construct the template URL using this pattern: https://<bucket-name>.s3.<region>.amazonaws.com/cfn-ecr-aws-soci-index-builder/templates/SociIndexBuilder.yml

Replace <bucket-name> with your S3 bucket name (starting with tcat-) and <region> with your deployment region.

Sign in to your AWS account, and open AWS Cloudformation console.
Choose the AWS Region you want to deploy and choose Create stack option, select With new resources(standard)
- Paste the S3 Url from the predeployment step
  
  Ensure that you select the same region that was specified in your .taskcat.yml file.
On the Specify stack details page, provide a stack name. Review the parameters for the template. Provide values for the parameters that require input, particularly the S3 bucket name (starting with tcat-) created during the taskcat upload step.

The CloudFormation template includes the following parameters:

SociIndexVersion: Determines the type of indices generated when a new image is pushed to the ECR repository.
- V2 (Recommended) - Latest version with enhanced security features
- V1 - Legacy version for backward compatibility
SociRepositoryImageTagFilters: Comma-separated list of SOCI repository image tag filters. Each filter is a repository name followed by a colon, ":" and followed by a tag. Both repository names and tags may contain wildcards denoted by an asterisk, "*". For example, prod/*:latest matches all images tagged with "latest" that are pushed to any repositories that start with "prod", while "dev:*" matches all images pushed to the "dev" repository. Use "*:*" to match all images pushed to all repositories in your private registry.
QSS3BucketName: Name of the S3 bucket for your copy of the deployment assets. Keep the default name unless you are customizing the template.
QSS3KeyPrefix: S3 key prefix that is used to simulate a folder for your copy of the deployment assets. Keep the default prefix unless you are customizing the template.

IamPermissionsBoundaryArn: IAM Roles might require an IAM Permissions boundary in order to be created and perform subsequent API calls to services. This parameter expects the ARN of an IAM policy, or to be set to none. When you finish reviewing and customizing the parameters, choose Next.

Unless you’re customizing the CFN template or are instructed otherwise in this guide’s Predeployment section, don’t change the default settings for the following parameters: QSS3BucketName, QSS3BucketRegion, and QSS3KeyPrefix. Changing the values of these parameters will modify code references that point to the Amazon Simple Storage Service (Amazon S3) bucket name and key prefix.

On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you finish, choose Next.
On the Review page, review and confirm the template settings. Under Capabilities, select all of the check boxes to acknowledge that the template creates AWS Identity and Access Management (IAM) resources that might require the ability to automatically expand macros.
Choose Create stack. The stack takes about 5 minutes to deploy.
Monitor the stack’s status, and when the status is CREATE_COMPLETE, the CFN AWS SOCI Index Builder deployment is ready.
To view the created resources, choose the Resources tab.

Cleanup

If you used automatic deployment and customized the stack_name field in your .taskcat.yaml file, the actual resource names will differ from those shown below, as they will use your custom stack name instead.

To remove this solution, delete the Amazon CloudFormation stack created in the Deployment steps section.
In the Amazon CloudWatch Logs console, delete the log groups /aws/lambda/<StackName>-SociIndexGeneratorLambda-<hash> and /aws/lambda/<StackName>-ECRImageActionEventFilteringLambda-<hash> where <StackName> is the name of the stack that you entered in the Deployment steps section. You can also use the DeleteLogGroup API to delete the log groups.
In the Amazon ECR console, delete the SOCI index artifacts. You can also use the BatchDeleteImage API to delete the artifacts.

Troubleshooting

For troubleshooting common issues, refer to Troubleshooting CloudFormation and the AWS Quick Start General Information Guide.

Lambda logs for generating SOCI index artifacts can be found in the CloudWatch log group /aws/lambda/<StackName>-SociIndexGeneratorLambda-<hash>. Lambda logs for repository image tag filtering and matches can be found in the CloudWatch log group /aws/lambda/<StackName>-ECRImageActionEventFilteringLambda-<hash>, where <StackName> is the name of the stack that you entered in the Deployment steps section.

If Lambda logs for generating SOCI index artifacts show that the Lambda function is timing out (that is, Task timed out is in the log), this is likely caused by the image size. SOCI Index Builder supports a maximum compressed image size of 6 GB because larger sizes may force the SOCI index generator Lambda to time out after 15 minutes.

A SOCI index will not be generated for an image if every layer in the image is under 10 MiB. For container images greater than 250 MiB (compressed), we recommend lazy loading. You are less likely to see a time reduction for smaller images.

Resources

Customer responsibility

After you deploy this template, confirm that your resources and services are updated and configured—including any required patches—to meet your security and other needs. For more information, refer to the Shared Responsibility Model.

Feedback

To submit feature ideas and report bugs, use the Issues section of the GitHub repository. To submit feedback on this deployment guide, use the following GitHub links:

Notices

This document is provided for informational purposes only. It represents current AWS product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS products or services, each of which is provided "as is" without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at https://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "as is" basis, without warranties or conditions of any kind, either expressed or implied. Refer to the License for specific language governing permissions and limitations.