DEV Community

Cover image for Building a Custom NGINX Module with Ansible: A Dev-Friendly Guide
Athreya aka Maneshwar
Athreya aka Maneshwar

Posted on

Building a Custom NGINX Module with Ansible: A Dev-Friendly Guide

#webdev #programming #beginners #devops

Hi there! I'm Maneshwar. Right now, I’m building LiveAPI, a first-of-its-kind tool that helps you automatically index API endpoints across all your repositories. LiveAPI makes it easier to discover, understand, and interact with APIs in large infrastructures.

NGINX is a powerful and versatile web server, and its extensibility through modules is one of its greatest strengths.

Sometimes, however, you need functionality that isn't available out-of-the-box or as a pre-compiled dynamic module.

This is where custom NGINX modules come in.

In this blog post, we'll walk through a real-world scenario: building NGINX with the ngx_http_consul_backend_module and the ngx_devel_kit (NDK) module.

We'll leverage Ansible to automate the entire process, making it repeatable, reliable, and dev-friendly.

Why Ansible for NGINX Builds?

Manually compiling NGINX with custom modules can be a tedious and error-prone process.

Dependencies, compilation flags, and directory structures all need to be precisely managed.

Ansible simplifies this significantly by:

  • Idempotency: Tasks can be run multiple times without causing unintended side effects.
  • Automation: Automate repetitive tasks, saving time and reducing human error.
  • Version Control: Store your build process in version control, enabling easy collaboration and rollbacks.
  • Consistency: Ensure consistent builds across different environments.

Our Goal: NGINX with Consul Backend Module

We'll be building NGINX 1.23.2 and integrating two specific modules:

  1. ngx_devel_kit (NDK): A collection of utilities and APIs that simplify the development of NGINX modules. Many custom modules rely on NDK.
  2. ngx_http_consul_backend_module: A module that allows NGINX to discover backend services from HashiCorp Consul. This is particularly useful in dynamic, microservices-based environments.

Project Structure

Our Ansible project is organized as a role named nginx-with-consul-module.

This structure promotes reusability and maintainability.

ansible
├─ README.md
├─ ansible.cfg
├─ hosts.ini
├─ install_ansible.sh
├─ nginx-build-playbook.yml
└─ roles
   └─ nginx-with-consul-module
      ├─ tasks
      │  ├─ build_consul_backend_module.yml
      │  ├─ build_nginx.yml
      │  ├─ configure_build.yml
      │  ├─ download_sources.yml
      │  ├─ install_dependencies.yml
      │  ├─ main.yml
      │  ├─ purge_deps.yml
      │  └─ systemd.yml
      └─ templates
         └─ nginx.service.j2
Enter fullscreen mode Exit fullscreen mode

The main.yml in the tasks directory orchestrates the entire build process by importing other task files:

---
- import_tasks: purge_deps.yml
- import_tasks: install_dependencies.yml
- import_tasks: download_sources.yml
- import_tasks: build_consul_backend_module.yml
- import_tasks: configure_build.yml
- import_tasks: build_nginx.yml
- import_tasks: systemd.yml
Enter fullscreen mode Exit fullscreen mode

Let's break down each step.

Step 1: Install Dependencies (install_dependencies.yml)

Building NGINX and its modules requires several development tools and libraries.

This task ensures all necessary packages are present on the target system.

---
- name: Install required packages
  apt:
    name:
      - build-essential # For gcc, g++ and make
      - libpcre3
      - libpcre3-dev # For PCRE regular expressions support
      - zlib1g
      - zlib1g-dev # For gzip compression support
      - libssl-dev # For OpenSSL (HTTPS) support
      - git # To clone the Consul backend module
      - wget # To download source archives
      - curl # General utility
    state: present
    update_cache: true
Enter fullscreen mode Exit fullscreen mode

Step 2: Download Sources (download_sources.yml)

Before we can compile anything, we need the source code for NGINX, NDK, and the Consul backend module.

We'll download these to a temporary directory.

---
- name: Create build directory
  file:
    path: /tmp
    state: directory

- name: Download nginx 1.23.2 source
  get_url:
    url: https://nginx.org/download/nginx-1.23.2.tar.gz
    dest: /tmp/nginx.tgz

- name: Extract nginx source
  unarchive:
    src: /tmp/nginx.tgz
    dest: /tmp/
    remote_src: yes

- name: Download NDK module
  get_url:
    url: https://github.com/simpl/ngx_devel_kit/archive/v0.3.0.tar.gz
    dest: /tmp/ngx_devel_kit-0.3.0.tgz

- name: Extract NDK module
  unarchive:
    src: /tmp/ngx_devel_kit-0.3.0.tgz
    dest: /tmp/
    remote_src: yes

- name: Clone Consul backend module
  git:
    repo: https://github.com/hashicorp/ngx_http_consul_backend_module.git
    dest: /go/src/github.com/hashicorp/ngx_http_consul_backend_module
Enter fullscreen mode Exit fullscreen mode

A crucial step here for the Consul backend module is a small patch.

The original code's strlen call with backend might cause a warning or error with newer compilers due to type mismatch.

We're explicitly casting backend to (const char*) to resolve this.

- name: Replace backend string length calculation
  replace:
    path: /go/src/github.com/hashicorp/ngx_http_consul_backend_module/src/ngx_http_consul_backend_module.c
    regexp: 'ngx_str_t ngx_backend = { strlen\(backend\), backend };'
    replace: "ngx_str_t ngx_backend = { strlen((const char*)backend), backend };"
Enter fullscreen mode Exit fullscreen mode

Step 3: Build Consul Backend Module (build_consul_backend_module.yml)

The ngx_http_consul_backend_module is written in Go and needs to be compiled as a C shared library. This involves several steps:

---
- name: Ensure nginx ext directory exists
  file:
    path: /usr/local/nginx/ext/
    state: directory
    owner: "{{ ansible_user | default('root') }}"
    group: "{{ ansible_user | default('root') }}"
    mode: "0755"

- name: Change ownership of Consul backend module directory
  file:
    path: /go/src/github.com/hashicorp/ngx_http_consul_backend_module
    state: directory
    owner: "{{ ansible_user | default('root') }}"
    group: "{{ ansible_user | default('root') }}"
    recurse: yes

- name: Check Go version
  shell: export PATH=/usr/local/go/bin:$PATH && /usr/local/go/bin/go version
  register: go_version_result
  changed_when: false
  failed_when: false

- name: Initialize Go modules
  command: /usr/local/go/bin/go mod init github.com/hashicorp/ngx_http_consul_backend_module
  args:
    chdir: /go/src/github.com/hashicorp/ngx_http_consul_backend_module
  register: go_mod_init_result
  changed_when: go_mod_init_result.rc == 0
  failed_when: go_mod_init_result.rc != 0 and "go.mod already exists" not in go_mod_init_result.stderr
  ignore_errors: true # Ignore if go.mod already exists

- name: Tidy Go modules
  command: /usr/local/go/bin/go mod tidy
  args:
    chdir: /go/src/github.com/hashicorp/ngx_http_consul_backend_module

- name: Print Go version
  debug:
    msg: "{{ go_version_result.stdout }}"

- name: Build Go shared library for Consul backend module
  shell: |
    export PATH=/usr/local/go/bin:$PATH
    CGO_CFLAGS="-I /tmp/ngx_devel_kit-0.3.0/src" \
    /usr/local/go/bin/go build -buildmode=c-shared -o /usr/local/nginx/ext/ngx_http_consul_backend_module.so ./src/ngx_http_consul_backend_module.go
  args:
    chdir: /go/src/github.com/hashicorp/ngx_http_consul_backend_module
Enter fullscreen mode Exit fullscreen mode

Key points:

  • We create a dedicated directory /usr/local/nginx/ext/ to store our compiled dynamic module.
  • We ensure correct ownership for the cloned Consul module directory.
  • We initialize and tidy Go modules to manage dependencies.
  • The go build -buildmode=c-shared command is crucial. It compiles the Go module into a C shared library (.so file) that NGINX can load dynamically.
  • CGO_CFLAGS="-I /tmp/ngx_devel_kit-0.3.0/src" is important because the Consul module depends on headers from the NDK, so we need to tell the Go compiler where to find them.

Step 4: Configure NGINX Build (configure_build.yml)

This step involves running the ./configure script for NGINX. This script generates the Makefile based on the desired modules and features.

# Simplified for brevity, but this is where you'd run ./configure
# with all your desired flags and --add-module directives.
# The original configure command from the user's prompt (commented out) would be used here.

# Example of how it would look if it were a separate task:
- name: Configure NGINX build
  command: >
    ./configure
    --prefix=/etc/nginx
    --sbin-path=/usr/sbin/nginx
    --conf-path=/etc/nginx/nginx.conf
    --pid-path=/var/run/nginx.pid
    --lock-path=/var/run/nginx.lock
    --error-log-path=/var/log/nginx/error.log
    --http-log-path=/var/log/nginx/access.log
    --with-http_ssl_module
    --with-http_stub_status_module
    --with-http_realip_module
    --with-http_auth_request_module
    --with-http_v2_module
    --with-http_dav_module
    --with-http_slice_module
    --with-http_addition_module
    --with-http_gunzip_module
    --with-http_gzip_static_module
    --with-http_sub_module
    --with-mail_ssl_module
    --with-stream_ssl_module
    --with-debug
    --add-module=/tmp/ngx_devel_kit-0.3.0
    --add-module=/go/src/github.com/hashicorp/ngx_http_consul_backend_module
  args:
    chdir: /tmp/nginx-1.23.2
Enter fullscreen mode Exit fullscreen mode

Important configure flags:

  • --prefix=/etc/nginx: Sets the installation prefix for NGINX.
  • --add-module=/path/to/module: This is critical for including our custom modules. We point to the extracted NDK and the cloned Consul backend module source directories.
  • Other --with-* flags enable various built-in NGINX modules like SSL, HTTP/2, etc.

Step 5: Build and Install NGINX (build_nginx.yml)

Once configure has generated the Makefile, we can proceed with compilation and installation.

---
- name: Compile nginx
  command: make
  args:
    chdir: /tmp/nginx-1.23.2

- name: Install nginx
  command: make install
  args:
    chdir: /tmp/nginx-1.23.2

- name: Install apache2-utils # Useful for htpasswd, etc.
  apt:
    name: apache2-utils
    state: present
Enter fullscreen mode Exit fullscreen mode
  • make: Compiles the NGINX source code along with the added modules.
  • make install: Installs NGINX and its components to the paths specified during the configure step.

Step 6: Systemd Integration (systemd.yml)

For a production-ready setup, we need NGINX to run as a system service.

This task typically involves creating a systemd service file.

# Example content for systemd.yml, assuming you have nginx.service.j2
---
- name: Copy nginx systemd service file
  template:
    src: nginx.service.j2
    dest: /etc/systemd/system/nginx.service
    owner: root
    group: root
    mode: '0644'
  notify: Reload systemd

- name: Enable nginx service
  systemd:
    name: nginx
    enabled: true
    daemon_reload: true # Ensures systemd picks up the new service file
    state: started

# Handler for 'Reload systemd'
# handlers/main.yml
# ---
# - name: Reload systemd
#   systemd:
#     daemon_reload: true
Enter fullscreen mode Exit fullscreen mode

The nginx.service.j2 template would define how NGINX starts, stops, and reloads as a systemd service.

Step 7: Cleanup (Optional but Recommended - purge_deps.yml)

After a successful build, you might want to remove temporary build files and even some build dependencies to free up space, especially in a containerized environment.

# Example content for purge_deps.yml
---
- name: Clean up temporary build files
  file:
    path: "{{ item }}"
    state: absent
  loop:
    - /tmp/nginx.tgz
    - /tmp/nginx-1.23.2
    - /tmp/ngx_devel_kit-0.3.0.tgz
    - /tmp/ngx_devel_kit-0.3.0
    - /go/src/github.com/hashicorp/ngx_http_consul_backend_module
Enter fullscreen mode Exit fullscreen mode

Running the Ansible Playbook

To execute this entire process, you would have a playbook like nginx-build-playbook.yml:

---
- name: Build NGINX with Consul module
  hosts: your_nginx_servers
  become: yes # Run tasks with sudo/root privileges
  roles:
    - nginx-with-consul-module
Enter fullscreen mode Exit fullscreen mode

And run it with:

ansible-playbook nginx-build-playbook.yml -i hosts.ini
Enter fullscreen mode Exit fullscreen mode

Conclusion

Building NGINX with custom modules can seem daunting, but by breaking down the process into logical Ansible tasks, we can create a robust, automated, and easily repeatable solution.

This approach is invaluable for consistent deployments across development, testing, and production environments.

You now have a solid foundation for integrating any custom NGINX module into your infrastructure using the power of Ansible!

LiveAPI helps you get all your backend APIs documented in a few minutes.

With LiveAPI, you can generate interactive API docs that allow users to search and execute endpoints directly from the browser.

LiveAPI Demo

If you're tired of updating Swagger manually or syncing Postman collections, give it a shot.

Top comments (4)

Collapse
 
dotallio profile image
Dotallio •

Love how you broke the process into clear Ansible roles, really makes custom NGINX builds less of a headache. How do you usually handle upgrades for these modules later on - have you automated that part as well?

Collapse
 
lovestaco profile image
Athreya aka Maneshwar •

Thanks

Collapse
 
parag_nandy_roy profile image
Parag Nandy Roy •

This is gold for DevOps beginners and pros alike ...

Collapse
 
lovestaco profile image
Athreya aka Maneshwar •

Haha so true