DEV Community

Cover image for Streaming PostgreSQL Backups with pgrwl: Now with Time & Count-Based Retention!
alexey.zh
alexey.zh

Posted on

Streaming PostgreSQL Backups with pgrwl: Now with Time & Count-Based Retention!

#devops #postgres #go #backup

A new release of pgrwl just shipped, a cloud-native WAL receiver and backup agent for PostgreSQL — and it comes with a new feature: streaming basebackups with automated retention.

What's New

feat(basebackup): time/count based retention

pgrwl can now:

  • Run basebackups on a schedule (via built-in cron),
  • Stream backups directly to remote storage (S3, SFTP, etc),
  • And automatically enforce retention policies:
    • Retain the last N backups (e.g. count=3)
    • Or retain all backups for a certain duration (e.g. days=7)

This makes it easier than ever to keep your PostgreSQL cluster safely backed up without relying on external scripts or schedulers.

Streaming Basebackup?

pgrwl performs a streaming basebackup:

  • Uses replication protocol
  • Streams backup with optional compression and encryption,
  • Uploads to remote storage as the backup progresses — no need for temporary local copies!

This design drastically reduces time-to-storage and fits perfectly into Kubernetes-native workflows.

Retention Policies in Action

Here's a typical config for scheduled backups:

backup:
  schedule: "0 3 * * *"  # every day at 3AM
  retention:
    type: count          # count-based retention
    count: 3             # keep last 3 backups
Enter fullscreen mode Exit fullscreen mode

After each successful backup, pgrwl automatically:

  • Lists available backups from storage,
  • Sorts them by timestamp,
  • Deletes old ones beyond the retention threshold — clean and simple.

Example: Kubernetes StatefulSet

A typical setup consists of two StatefulSets:

  1. Receiver — continuously streams and archives WALs.
  2. Backup — schedules full basebackups and handles retention.

On restore, the receiver pod can switch to serve mode, exposing archived WALs over HTTP to your PostgreSQL’s restore_command.

Why This Matters

For teams running PostgreSQL in Kubernetes (or other containerized environments), backup tooling is often a headache:

  • No clear story for WAL + basebackup orchestration,
  • Manual scripts with little observability,
  • Poor retention handling.

pgrwl aims to solve all of that, with:

  • Streaming basebackups
  • Streaming WAL receiver
  • Pluggable storage
  • Built-in compression/encryption
  • Clean Kubernetes integration
  • Minimal dependencies

Try It Out

Get started here 👉 https://github.com/hashmap-kz/pgrwl

If you have feedback, issues, or want to contribute, we’d love to hear from you! 🙌

Top comments (0)