DEV Community

Purushotam Adhikari
Purushotam Adhikari

Posted on

Automating Linux Updates Across Multiple Servers with Ansible

#ansible #automation #devops #infrastructure

Managing updates across multiple Linux servers can be a time-consuming and error-prone task when done manually. Ansible provides an elegant solution to automate this process, ensuring consistency, reducing human error, and saving valuable time. In this comprehensive guide, we'll explore how to set up automated Linux updates using Ansible, covering everything from basic configurations to advanced strategies.

Why Automate Server Updates?

Before diving into the technical implementation, let's understand why automation is crucial:

  • Consistency: Ensures all servers receive the same updates in the same manner
  • Time Efficiency: Eliminates the need to SSH into each server individually
  • Reduced Human Error: Minimizes mistakes that can occur during manual updates
  • Scheduling: Allows updates during maintenance windows
  • Logging and Reporting: Provides detailed logs of what was updated and when
  • Rollback Capabilities: Enables quick recovery if issues arise

Prerequisites

To follow this guide, you'll need:

  • Ansible installed on your control machine
  • SSH access to target servers
  • Sudo privileges on target servers
  • Basic understanding of YAML syntax
  • Target servers running supported Linux distributions (Ubuntu, CentOS, RHEL, Debian)

Setting Up Your Ansible Environment

1. Install Ansible

On Ubuntu/Debian:

sudo apt update
sudo apt install ansible
Enter fullscreen mode Exit fullscreen mode

On CentOS/RHEL:

sudo yum install epel-release
sudo yum install ansible
Enter fullscreen mode Exit fullscreen mode

2. Configure SSH Key Authentication

Generate SSH keys and copy them to your target servers:

ssh-keygen -t rsa -b 4096
ssh-copy-id [email protected]
ssh-copy-id [email protected]
Enter fullscreen mode Exit fullscreen mode

3. Create Your Inventory File

Create an inventory file (hosts.yml) to define your servers:

all:
  children:
    production:
      hosts:
        web-server-1:
          ansible_host: 192.168.1.10
          ansible_user: ubuntu
        web-server-2:
          ansible_host: 192.168.1.11
          ansible_user: ubuntu
        db-server-1:
          ansible_host: 192.168.1.20
          ansible_user: centos
    staging:
      hosts:
        staging-web:
          ansible_host: 192.168.1.50
          ansible_user: ubuntu
    development:
      hosts:
        dev-server:
          ansible_host: 192.168.1.60
          ansible_user: ubuntu
Enter fullscreen mode Exit fullscreen mode

Basic Update Playbook

Let's start with a simple playbook that updates all packages on Ubuntu/Debian systems:

---
- name: Update Linux servers
  hosts: all
  become: yes
  gather_facts: yes

  tasks:
    - name: Update apt cache (Ubuntu/Debian)
      apt:
        update_cache: yes
        cache_valid_time: 3600
      when: ansible_os_family == "Debian"

    - name: Upgrade all packages (Ubuntu/Debian)
      apt:
        upgrade: dist
        autoremove: yes
        autoclean: yes
      when: ansible_os_family == "Debian"
      register: apt_upgrade_result

    - name: Update yum cache (CentOS/RHEL)
      yum:
        update_cache: yes
      when: ansible_os_family == "RedHat"

    - name: Upgrade all packages (CentOS/RHEL)
      yum:
        name: "*"
        state: latest
      when: ansible_os_family == "RedHat"
      register: yum_upgrade_result

    - name: Check if reboot is required (Ubuntu/Debian)
      stat:
        path: /var/run/reboot-required
      register: reboot_required_file
      when: ansible_os_family == "Debian"

    - name: Display upgrade results
      debug:
        msg: "{{ apt_upgrade_result.stdout_lines if ansible_os_family == 'Debian' else yum_upgrade_result.results }}"
Enter fullscreen mode Exit fullscreen mode

Save this as update-servers.yml and run it with:

ansible-playbook -i hosts.yml update-servers.yml
Enter fullscreen mode Exit fullscreen mode

Advanced Update Strategies

1. Rolling Updates with Error Handling

For production environments, you'll want to update servers in batches to maintain service availability:

---
- name: Rolling server updates
  hosts: production
  become: yes
  gather_facts: yes
  serial: 2  # Update 2 servers at a time
  max_fail_percentage: 10  # Stop if more than 10% fail

  pre_tasks:
    - name: Check server connectivity
      ping:

    - name: Verify disk space
      shell: df -h / | awk 'NR==2 {print $5}' | sed 's/%//'
      register: disk_usage
      failed_when: disk_usage.stdout|int > 90

  tasks:
    - name: Create backup directory
      file:
        path: /backup/pre-update-{{ ansible_date_time.date }}
        state: directory
        mode: '0755'

    - name: Backup package list (Ubuntu/Debian)
      shell: dpkg --get-selections > /backup/pre-update-{{ ansible_date_time.date }}/packages.list
      when: ansible_os_family == "Debian"

    - name: Backup package list (CentOS/RHEL)
      shell: rpm -qa > /backup/pre-update-{{ ansible_date_time.date }}/packages.list
      when: ansible_os_family == "RedHat"

    - name: Update package cache
      package:
        update_cache: yes
      retries: 3
      delay: 5

    - name: Upgrade all packages
      package:
        name: "*"
        state: latest
      register: upgrade_result
      notify:
        - Check if reboot required

    - name: Remove unused packages (Ubuntu/Debian)
      apt:
        autoremove: yes
        purge: yes
      when: ansible_os_family == "Debian"

  handlers:
    - name: Check if reboot required
      stat:
        path: /var/run/reboot-required
      register: reboot_required
      notify: Conditional reboot

    - name: Conditional reboot
      reboot:
        reboot_timeout: 300
        pre_reboot_delay: 5
      when: reboot_required.stat.exists | default(false)

  post_tasks:
    - name: Verify services are running
      service:
        name: "{{ item }}"
        state: started
      loop:
        - ssh
        - cron
      ignore_errors: yes

    - name: Send notification
      mail:
        to: [email protected]
        subject: "Server {{ inventory_hostname }} updated successfully"
        body: "Updates completed at {{ ansible_date_time.iso8601 }}"
      when: upgrade_result.changed
      delegate_to: localhost
Enter fullscreen mode Exit fullscreen mode

2. Scheduled Updates with Maintenance Windows

Create a playbook that respects maintenance windows:

---
- name: Scheduled maintenance updates
  hosts: all
  become: yes
  gather_facts: yes

  vars:
    maintenance_start: "02:00"
    maintenance_end: "04:00"
    current_time: "{{ ansible_date_time.hour }}:{{ ansible_date_time.minute }}"

  tasks:
    - name: Check if we're in maintenance window
      set_fact:
        in_maintenance_window: "{{ (current_time >= maintenance_start) and (current_time <= maintenance_end) }}"

    - name: Skip updates outside maintenance window
      debug:
        msg: "Skipping updates - outside maintenance window ({{ maintenance_start }} - {{ maintenance_end }})"
      when: not in_maintenance_window

    - name: Proceed with updates
      block:
        - name: Update repositories
          package:
            update_cache: yes

        - name: Install security updates only
          package:
            name: "*"
            state: latest
            security: yes
          when: ansible_os_family == "RedHat"

        - name: Install unattended-upgrades for security updates (Ubuntu/Debian)
          apt:
            name: unattended-upgrades
            state: present
          when: ansible_os_family == "Debian"

      when: in_maintenance_window
Enter fullscreen mode Exit fullscreen mode

3. Selective Updates with Package Exclusions

Sometimes you need to exclude certain packages from updates:

---
- name: Selective package updates
  hosts: all
  become: yes
  gather_facts: yes

  vars:
    excluded_packages:
      - kernel*
      - docker*
      - mysql*

  tasks:
    - name: Update all packages except excluded ones (Ubuntu/Debian)
      apt:
        upgrade: safe
        update_cache: yes
      when: ansible_os_family == "Debian"

    - name: Hold excluded packages (Ubuntu/Debian)
      dpkg_selections:
        name: "{{ item }}"
        selection: hold
      loop: "{{ excluded_packages }}"
      when: ansible_os_family == "Debian"

    - name: Update non-excluded packages (CentOS/RHEL)
      yum:
        name: "*"
        state: latest
        exclude: "{{ excluded_packages | join(',') }}"
      when: ansible_os_family == "RedHat"
Enter fullscreen mode Exit fullscreen mode

Monitoring and Reporting

Update Status Report Playbook

Create a comprehensive reporting system:

---
- name: Generate update report
  hosts: all
  become: yes
  gather_facts: yes

  tasks:
    - name: Check for available updates (Ubuntu/Debian)
      shell: apt list --upgradable 2>/dev/null | grep -v "WARNING" | wc -l
      register: available_updates_debian
      when: ansible_os_family == "Debian"
      changed_when: false

    - name: Check for available updates (CentOS/RHEL)
      shell: yum check-update | grep -E "^[a-zA-Z]" | wc -l
      register: available_updates_redhat
      when: ansible_os_family == "RedHat"
      changed_when: false
      failed_when: false

    - name: Check last update time
      stat:
        path: /var/log/apt/history.log
      register: apt_history
      when: ansible_os_family == "Debian"

    - name: Check system uptime
      shell: uptime -s
      register: system_uptime
      changed_when: false

    - name: Generate report
      template:
        src: update_report.j2
        dest: /tmp/update_report_{{ inventory_hostname }}.txt
      delegate_to: localhost
      vars:
        available_updates: "{{ available_updates_debian.stdout if ansible_os_family == 'Debian' else available_updates_redhat.stdout }}"
        last_boot: "{{ system_uptime.stdout }}"
Enter fullscreen mode Exit fullscreen mode

Create a Jinja2 template (update_report.j2):

Server Update Report
===================
Server: {{ inventory_hostname }}
OS: {{ ansible_distribution }} {{ ansible_distribution_version }}
Architecture: {{ ansible_architecture }}
Last Boot: {{ last_boot }}
Available Updates: {{ available_updates }}
Generated: {{ ansible_date_time.iso8601 }}

{% if available_updates|int > 0 %}
WARNING: {{ available_updates }} updates available
{% else %}
Status: System up to date
{% endif %}

System Information:
- Memory: {{ ansible_memtotal_mb }}MB
- CPU Cores: {{ ansible_processor_vcpus }}
- Disk Usage: {{ ansible_mounts[0].size_available }} bytes available
Enter fullscreen mode Exit fullscreen mode

Best Practices and Tips

1. Testing Strategy

Always test your playbooks in a development environment first:

# Test syntax
ansible-playbook --syntax-check update-servers.yml

# Dry run
ansible-playbook -i hosts.yml update-servers.yml --check

# Run on development servers first
ansible-playbook -i hosts.yml update-servers.yml --limit development
Enter fullscreen mode Exit fullscreen mode

2. Backup Strategy

Always create backups before major updates:

- name: Create system snapshot (if using LVM)
  shell: lvcreate -L1G -s -n snapshot-{{ ansible_date_time.date }} /dev/vg0/root
  when: ansible_lvm is defined
  ignore_errors: yes
Enter fullscreen mode Exit fullscreen mode

3. Logging and Auditing

Configure comprehensive logging:

- name: Log update activity
  lineinfile:
    path: /var/log/ansible-updates.log
    line: "{{ ansible_date_time.iso8601 }} - Updates applied by {{ ansible_user_id }}"
    create: yes
Enter fullscreen mode Exit fullscreen mode

4. Error Handling

Implement robust error handling:

tasks:
  - name: Update packages
    package:
      name: "*"
      state: latest
    register: update_result
    failed_when: false

  - name: Handle update failures
    debug:
      msg: "Update failed on {{ inventory_hostname }}: {{ update_result.msg }}"
    when: update_result.failed

  - name: Continue with next server
    meta: clear_host_errors
    when: update_result.failed
Enter fullscreen mode Exit fullscreen mode

Automation with Cron

Set up automated execution using cron:

# Add to crontab for weekly updates
0 2 * * 0 /usr/bin/ansible-playbook -i /path/to/hosts.yml /path/to/update-servers.yml >> /var/log/ansible-cron.log 2>&1
Enter fullscreen mode Exit fullscreen mode

Conclusion

Automating Linux updates with Ansible provides numerous benefits including consistency, reliability, and time savings. The examples provided in this guide offer a solid foundation that you can customize based on your specific requirements.

Key takeaways:

  • Start with simple playbooks and gradually add complexity
  • Always test in development environments first
  • Implement proper backup and rollback strategies
  • Use rolling updates for production environments
  • Monitor and log all update activities
  • Respect maintenance windows and business requirements

Remember that automation is not set-and-forget. Regularly review and update your playbooks, monitor their execution, and stay informed about security advisories for your systems.

The investment in setting up automated updates pays dividends in reduced manual work, improved security posture, and more reliable infrastructure management.

Have you implemented automated updates in your environment? Share your experiences and tips in the comments below!

Resources

Top comments (0)