Adding Self-Hosted Agents

Codeberg's CI resources are run by volunteers, and as such have some limitations in terms of shared resources. However, it is not necessary to self-host and administer your own full CI instance as you can add your own Woodpecker agents to your user or organization.

Some example use-cases where a self-hosted agent are recommended include: when you want faster feedback from CI, to handle long-running or resource-intensive jobs, creating specialized environments for specific architectures, hardware-in-the-loop testing, or setting up larger networked integration tests.

This guide will walk you through setting up your own Woodpecker agent to communicate with Codeberg's CI infrastructure and configuring your workflows to use it.

Obtaining an agent token

Before deploying the agent, you need to obtain an agent token from Codeberg's Woodpecker instance. The agent token is a secret shared between Codeberg's CI instance and your agent in order to facilitate the execution of workflows.

To do so, please follow the instructions immediately below for users, and follow the instructions below that for organizations.

1. Go to the agents page

   1. For users, go to the Codeberg Woodpecker CI User Agent settings

   2. For organizations:

      1. Go to the Codeberg Woodpecker CI Repos Page

      2. Click on a repository in the organization you're adding the agent to.

      3. Click on the organization name in the top left.

      4. Click on the settings icon to access the organization settings in the top right.

      5. Click on the agents tab.

2. Click add agent.

3. Type in a descriptive or memorable name. This can be changed later.

4. Click the green add agent button at the bottom of the form.

5. Copy the grey token from the token textbox.

This token will be used to set up the Woodpecker Agent in the next step.

Creating a Woodpecker Agent

The Woodpecker Agent is released as a docker image and as Linux distribution packages. The installation is simplified for Docker Compose and Helm deployments.

The following configurations are modified to work with Codeberg's CI service from the official documentation. For more information on the specific variables available to configure, please read the Woodpecker agent documentation.

Creating an agent with Docker Compose

services:
  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:v3
    command: agent
    restart: always
    volumes:
      - woodpecker-agent-config:/etc/woodpecker
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=grpc.ci.codeberg.org
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
      - WOODPECKER_GRPC_SECURE=true

volumes:
  woodpecker-agent-config:

Automatically scaling your agents on demand

Woodpecker also has a container to automatically scale agents. This is not required for Kubernetes clusters. Please read the autoscaler documentation for more information

Managing agents with Kubernetes and Helm

Alternatively, Kubernetes can aid in managing your agents. It provides automatic scaling, metrics, topology management and whole host of other features that can be tuned for a particular project. For more details, please see the Helm chart for Woodpecker Agents.

Only the agent chart is required.

helm install woodpecker/agent oci://ghcr.io/woodpecker-ci/helm/woodpecker --version <VERSION>

Configuring workflows to use your agent

Now that you have added your agent to Codeberg CI, you can configure your workflows to use your new agent. This is recommended for resource intensive and long running jobs. This can also be used to create specialized agents that have access to hardware or other environmental configurations.

For example, if you have a workflow you want to run on ARM and RISCV architecture, these require adding your own agent (as of the time of writing). You can use the following workflow as an example. For the specifics of the syntax, please read the the Woodpecker Docs. The platforms are written as Go platforms:

matrix:
  DATABASE:
    - postgres
    - mariadb
  platform:
    - darwin/arm64
    - linux/amd64
    - linux/arm64
    - linux/riscv64

labels:
  platform: ${platform}

steps:
  # run tests on all platforms
  - name: test
    image: rust
    commands:
      - cargo test

  # check benchmarks on amd64
  - name: test-benchmarks-amd64
    image: rust
    commands:
      - cargo bench
    when:
      platform: */amd64

  # check benchmarks on arm64
  - name: test-benchmarks-arm64
    image: rust
    commands:
      - cargo bench
    when:
      platform: */arm64

  # check benchmarks on riscv64
  - name: test-benchmarks-riscv64
    image: rust
    commands:
      - cargo bench
    when:
      platform: */riscv64

You can also add labels to the agent's configuration that can be used in workflows. Labels are key-value maps assigned to specific agents by using the WOODPECKER_AGENT_LABELS variable.

For example, say you're creating an IoT project that needs a machine with a specific peripheral, such as a LoRa radio, for testing. You have an agent that has the module attached and you need a way to tell Woodpecker to use that particular machine. Radio hardware often depends on what part of the world you're in, you also want specify where your agent is located.

Here is how to add the necessary environment variable to an agent's docker compose script to label it in ITU Region 1 (Europe is in Region 1) with a lora module attached:

# Agent configuration goes here...
environment:
  - WOODPECKER_AGENT_LABELS=location=region-1,peripheral=lora
# Other environment variables go here...

You then use the following workflow to specify tests for those labels:

labels:
  # Only select agents with a location of `region-1`...
  location: region-1
  # ...and with the peripheral label set to `lora`
  peripheral: lora

steps:
  - name: test-debian
    image: debian
    commands:
      - apt install build-essentials
      - make build
      - make test

If you had a teammate running an agent in another continent, you would then change the location label in both the workflow and the agent from region-1 to region-2 or region-3 accordingly. If you wanted to run tests with a different peripheral, you'd change the peripheral label, etc.

References

• Woodpecker General Administration Guide
• Woodpecker Docker Compose Installation Guide
• Woodpecker Helm Installation Guide
• Woodpecker Workflows Reference
• Woodpecker Workflow Syntax Reference