How to Test and Reduce Duplications in GitHub Actions Workflows

If you are seeing duplications in your pipeline and are looking for a way of reusing either part or whole of that workflow, wondering how you would variablise your current workflow, you are at the right place! I've encountered similar issues before and going to walk you through the steps so you can manage your pipelines to reduce the likelihood of human error. GitHub has the steps well documented, this article is both for my own benefit to highlight the trial and error I've been through.

There are in general three ways of refactoring your GitHub actions

• Composite actions
• Reusable workflows
• Matrix strategy

As an overview, in terms of size of repeated action YAML chunks reused from workflows, from smallest to largest: composite actions (variates action steps), matrix strategy (variates action jobs), reusable workflows (variates workflows).

Composite actions

In my own use case, I wanted to run the same test commands for all my modules, this means that I have to repeat the same set of steps under the run test job. I could use a matrix test setup but that would duplicate the compilations because the matrix runs at per job level, it would repetitively run the installations for each test. I want to have variations for the set of job steps, such that I can pass an input which will be the name of the module to execute the steps on. Here's a snippet you can copy as a template, referenced from Github documentation, it takes an input and output (both are optional):

The composite action:

name: 'Hello World'
description: 'Greet someone'
inputs:
  who-to-greet:  # id of input
    description: 'Who to greet'
    required: true
    default: 'World'
outputs:
  random-number:
    description: "Random number"
    value: ${{ steps.random-number-generator.outputs.random-number }}
runs:
  using: "composite"
  steps:
    - name: Set Greeting
      run: echo "Hello $INPUT_WHO_TO_GREET."
      shell: bash
      env:
        INPUT_WHO_TO_GREET: ${{ inputs.who-to-greet }}

    - name: Random Number Generator
      id: random-number-generator
      run: echo "random-number=$(echo $RANDOM)" >> $GITHUB_OUTPUT
      shell: bash

The caller of the composition action:

on: [push]

jobs:
  hello_world_job:
    runs-on: ubuntu-latest
    name: A job to say hello
    steps:
      - uses: actions/checkout@v4
      - id: foo
        uses: OWNER/hello-world-composite-action@SHA
        with:
          who-to-greet: 'Mona the Octocat'
      - run: echo random-number "$RANDOM_NUMBER"
        shell: bash
        env:
          RANDOM_NUMBER: ${{ steps.foo.outputs.random-number }}

Hints

File structure

You must have an action.yaml, action.yml or Dockerfile file nested under your specified action name, other you will hit error

Error: Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/.github/actions/test-composite-action'. Did you forget to run actions/checkout before running your local action?

The folder structure for your workflows should look something like this

my-app/
├─ .github/
│  ├─ workflows/
│  │  ├─ my-composite-action/
│  │  │  ├─ action.yaml
│  │  ├─ my-workflow

Specify shell

If you have run within your step, it is compulsory to add a shell property within a composite action shell: bash, otherwise you will hit the error:

Required property is missing: shell

Matrix Strategy

This is useful if you want to run variations of a job within a workflow, all unique combinations of the variables will be executed for the job, as an example the following creates 6 jobs, taken from Github documentation:

  example_matrix:
    strategy:
      matrix:
        version: [10, 12, 14]
        os: [ubuntu-latest, windows-latest]

One big pro of the matrix strategy approach is that each combination is run in parallel, whereas composite actions are run sequentially. However matrix strategy has less reusability compared to composite actions, because it is tied to the specific workflow job.

Reusable workflows

This is generally desirable if the same workflow has to be triggered for multiple repositories, in which case you don't want to copy and paste the same workflows for each repositories, this will introduce extra maintenance overheads. Instead you should consider having a central repository to store the reusable workflows. Examples of good candidates for reusable workflows are testing and formatting. Here's a snippet of a reusable workflow which takes an input and a secret, both optional (taken from Github Documentation):

# workflow-B.yml
name: Reusable workflow example

on:
  workflow_call:
    inputs:
      config-path:
        required: true
        type: string
    secrets:
      token:
        required: true

jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/labeler@v4
      with:
        repo-token: ${{ secrets.token }}
        configuration-path: ${{ inputs.config-path }}

Caller of the reusable workflow:

name: Call a reusable workflow

on:
  pull_request:
    branches:
      - main

jobs:
  call-workflow-passing-data:
    permissions:
      contents: read
      pull-requests: write
    uses: octo-org/example-repo/.github/workflows/workflow-B.yml@main
    with:
      config-path: .github/labeler.yml
    secrets:
      token: ${{ secrets.GITHUB_TOKEN }}

How to test workflows

The most obvious way to test the workflows is to create a dummy change that will trigger the workflow, however this does not work if your workflow only runs on push to main. Even if you added a trigger temporarily for pull requests, this still doesn't fully replicate the environment e.g. commits on PR are squashed into one by the checkout action. It also pollutes the commit history on pull requests with dummy test commits. An easier way is to add

on:
  workflow_dispatch:

to your workflow. This then allows you to trigger the workflow manually. Navigate to the Actions tab, find your workflow, once you click on it you will see a Run workflow button with an option of the branch you want to run it on.