AMIgo

AMIgo is an application for baking AMIs (Amazon Machine Images). For information on how to use Amigo baked AMIs with Riffraff check here. This project is built in teamcity.

Terminology

• A base image is the source AMI to use as the basis for an image. For example you might have an "Ubuntu Wily" base image.

• A role is something installed or configured on the machine. For example if you want your machine to have a JVM, Node and nginx pre-installed, you would assign the corresponding roles to your image. Currently roles are implemented as Ansible roles.

• A recipe is a description of how to bake your AMI. Making a recipe consists of choosing a base image and deciding which roles to assign. For example you might have a recipe that builds an image based on Ubuntu Wily and installs a JVM, Node and nginx.

• A bake is a single execution of a recipe. The result of a bake is an AMI.

Implementation

AMIgo is implemented as a Play application. It uses Packer and Ansible to bake AMIs.

AMI baking process

Roughly, AMIgo does the following:

1. Dynamically generate an Ansible playbook based on the recipe's roles
2. Dynamically generate a Packer build configuration file to install and then run Ansible
3. Execute Packer as an external process
4. Parse the Packer output and extract useful information from it

All data (base images, recipes, bakes, bake logs) are stored in DynamoDB. The Dynamo tables are created automatically if they do not exist.

Debugging recipes

When running in an environment that is not PROD there is an option to Bake with debug enabled. This passes the -debug flag through to packer which saves a copy of the SSH key in AMIgos working directory. This makes it possible to SSH onto the instance that is being used to build the AMI.

How to run locally

Load the deployTools credentials using Janus, then execute ./script/server. This will run the Amigo app locally and the associated packer process should have the sufficient AWS authorization.

$ cat ~/.configuration-magic/amigo.conf

packer {
  vpcId = "vpc-1234abcd"
  subnetId = "subnet-5678efgh"
  instanceProfile = "[optional] instance profile name for the box packer will run on"
}

ansible {
  packages {
    s3bucket = "your-bucket"
    s3prefix = "an/optional/prefix/"
  }
}

$ sbt run

How to run the tests

$ sbt test

Required AWS permissions for Packer

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AttachVolume",
                "ec2:CreateVolume",
                "ec2:DeleteVolume",
                "ec2:CreateKeypair",
                "ec2:DeleteKeypair",
                "ec2:DescribeSubnets",
                "ec2:CreateSecurityGroup",
                "ec2:DeleteSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateImage",
                "ec2:CopyImage",
                "ec2:RunInstances",
                "ec2:TerminateInstances",
                "ec2:StopInstances",
                "ec2:DescribeVolumes",
                "ec2:DetachVolume",
                "ec2:DescribeInstances",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot",
                "ec2:DescribeSnapshots",
                "ec2:DescribeImages",
                "ec2:RegisterImage",
                "ec2:CreateTags",
                "ec2:ModifyImageAttribute",
                "iam:*",
                "elasticloadbalancing:*"
            ],
            "Resource": "*"
        }
    ]
}

Testing ansible scripts without runing amigo/packer

Tired of waiting for amigo to build, deploy and bake only to discover you made a one character error in your ansible script? Then read on...

You can use Vagrant to test ansible scripts. Once set up, it allows you to try out your script with a feedback loop of 20 seconds or so. There are some docs here covering this, but, roughly speaking you need to:

Pre-requisites

1. Vagrant
2. Virtualbox

Running Ansible roles

1. execute ./script/setup
2. cd into ansible/
3. update files playbook.yaml and extra-vars.yaml accordingly
4. run vagrant up to download the image and run your ansible script
5. run vagrant provision to re-run the ansible script