Automating AWS Infrastructure Provisioning with CloudFormation and GitLab CI/CD

#aws #gitlab #cloudformation #cicd

Provisioning infrastructure using Infrastructure as Code (IaC) brings repeatability, scalability, and automation to cloud deployments. In this blog post, we'll walk through how to provision a simple AWS web server using CloudFormation templates and GitLab CI/CD pipelines, targeting three environments — development, staging, and production.
In previous posts, we have implemented similar AWS infrastructure using CodePipeline, Jenkins, GitSync, and now we will use GitLab CI.

Why GitLab CI/CD for AWS Infrastructure Provisioning?

GitLab CI/CD offers a seamless DevOps workflow, allowing you to:

Integrate Infrastructure as Code directly into your version-controlled repository.
Automate infrastructure provisioning on merge or tag events.
Use secure environment variables to manage AWS credentials.
Promote infrastructure changes through different environments using Git branches or tags.

GitLab Pipelines help remove human errors and ensure consistent, validated deployments every time code is committed.

Architecture Overview

We’re going to deploy a simple web server (EC2 instance with Apache) on AWS using CloudFormation. Each environment (development, staging, production) will have:

Step 1: Prerequisites

Before setting up the pipeline, you’ll need:

GitLab CI/CD Variables
Head to your GitLab Project → Settings → CI/CD → Variables and set the following:

AWS_ACCESS_KEY_ID Your AWS Access Key ID Masked , Protected
AWS_SECRET_ACCESS_KEY Your AWS Secret Access Key Masked, Protected
AWS_DEFAULT_REGION e.g., us-east-1 Not Masked, Protected

Use a user with appropriate permissions. For this example, we've used an administrator-level IAM user for simplicity. In real-world scenarios, prefer least-privilege IAM roles and policies.

Step 2: Directory Structure and Templates

We will organize our repository with separate folders per environment, and use nested stacks to split infrastructure into logical components. Please refer to GitLab repo for more details.

Each root.yaml template includes network.yaml and compute.yaml as nested stacks.

.gitlab-ci.yml
infrastructure/
├── development/
│   └── root.yaml
│   └── network.yaml
│   └── compute.yaml
├── staging/
│   └── root.yaml
│   └── network.yaml
│   └── compute.yaml
└── production/
    └── root.yaml
    └── network.yaml
    └── compute.yaml

Step 3: GitLab CI/CD Pipeline (.gitlab-ci.yml)

Pipeline Steps:

create_bucket (Stage: s3_repository): Deploys an S3 bucket using a CloudFormation template to store other templates.
copy_templates (Stage: s3_repository): Uploads all CloudFormation templates from the repo to the S3 bucket.
lint_templates (Stage: lint): Uses python:3.11 image to install and run cfn-lint on all .yaml templates for syntax and structure validation.
validate_templates (Stage: validate): Runs aws cloudformation validate-template on each environment's root template via S3 URLs to ensure correctness.
deploy_dev (Stage: deploy): Deploys the development stack using the development/root.yaml template if on the main branch.
deploy_staging (Stage: deploy): Deploys the staging stack similarly using staging/root.yaml.
deploy_production (Stage: deploy): Manual trigger to deploy the production stack using production/root.yaml.

image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest

stages:
  - s3_repository
  - lint
  - validate
  - deploy

variables:
  AWS_REGION: us-east-1
  BUCKET_NAME: ct-cfn-files-for-stack

before_script:
  - aws sts get-caller-identity

create_bucket:
  stage: s3_repository
  script:
    - ls -l infrastructure
    - echo "Creating S3 bucket for CloudFormation templates..."
    - |
      aws cloudformation deploy \
        --stack-name cfn-template-bucket \
        --template-file infrastructure/create-cfn-template-bucket.yaml \
        --region ${AWS_REGION} \
        --capabilities CAPABILITY_NAMED_IAM

copy_templates:
  needs: ["create_bucket"]
  stage: s3_repository
  script:
    - echo "Copying CloudFormation templates to S3 bucket..."
    - aws s3 cp infrastructure/ s3://${BUCKET_NAME}/infrastructure/ --recursive

lint_templates:
  image: python:3.11
  stage: lint
  before_script:
    - pip install cfn-lint
  script:
    - echo "Linting CloudFormation templates..."
    - |
      ERR=0
      for file in $(find ./infrastructure -type f \( -iname "*.yaml" -o -iname "*.yml" \)); do
        cfn-lint "$file" || ERR=1
      done
      if [ "$ERR" -eq "1" ]; then
        exit 1
      fi

validate_templates:
  stage: validate
  script:
    - echo "Validating CloudFormation templates..."
    - aws cloudformation validate-template --template-url https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com/infrastructure/development/root.yaml
    - aws cloudformation validate-template --template-url https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com/infrastructure/staging/root.yaml
    - aws cloudformation validate-template --template-url https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com/infrastructure/production/root.yaml

deploy_dev:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  script:
    - echo "Development Deployment..."
    - |
      aws cloudformation create-stack \
        --template-url https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com/infrastructure/development/root.yaml \
        --stack-name DeployDevelopmentStack \
        --parameters ParameterKey=Environment,ParameterValue=development \
        --capabilities CAPABILITY_NAMED_IAM
    - aws cloudformation wait stack-create-complete --stack-name DeployDevelopmentStack

deploy_staging:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  script:
    - echo "Staging Deployment..."
    - |
      aws cloudformation create-stack \
        --template-url https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com/infrastructure/staging/root.yaml \
        --stack-name DeployStagingStack \
        --parameters ParameterKey=Environment,ParameterValue=staging \
        --capabilities CAPABILITY_NAMED_IAM
    - aws cloudformation wait stack-create-complete --stack-name DeployStagingStack

deploy_production:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual
  script:
    - echo "Production Deployment..."
    - |
      aws cloudformation create-stack \
        --template-url https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com/infrastructure/production/root.yaml \
        --stack-name DeployProductionStack \
        --parameters ParameterKey=Environment,ParameterValue=production \
        --capabilities CAPABILITY_NAMED_IAM
    - aws cloudformation wait stack-create-complete --stack-name DeployProductionStack

Step 4: Git Push to repo to see automated GitLab pipeline running

Git push the components to GitLab Repo.

Development and Staging environments getting provisioned whereas production environment stage awaiting manual approval.

Manual approval provided:

Pipeline logs:

Cleanup

Don’t forget to delete AWS resources to avoid unexpected costs:

Use aws cloudformation delete-stack --stack-name for each stack.
Optionally delete the S3 bucket if no longer needed.

Conclusion

In this post, we demonstrated how to provision AWS infrastructure using CloudFormation with GitLab CI/CD. With this setup, you can manage your infrastructure through code, version control changes, and ensure your environments are consistent and reproducible. GitLab Pipelines make it easy to push updates across environments with minimal manual effort.

By combining the power of CloudFormation, GitLab, and S3, you’ve built a scalable and secure workflow to deploy cloud resources.