DEV Community

邱敬幃 Pardn Chiu
邱敬幃 Pardn Chiu

Posted on • Edited on

Redis Fallback (Golang)

cover

Redis Fallback (Golang)

A Redis fallback package that automatically degrades to local storage, ensuring minimal data loss during fallback and seamless recovery when Redis becomes available again.

Extended from the integration concept of php-redis, php-cache-fallback, and php-session-fallback, providing a unified solution for Golang.

Three key features

  • Three-tier Storage Architecture: Memory cache + Redis + Local file storage with automatic fallback
  • Automatic Recovery: Periodically monitors Redis health status and batch synchronizes data after recovery
  • Data Persistence: Stores data as JSON files during fallback mode to prevent data loss with TTL support

Flow

Click to show

flowchart TD
  A[Initialize] --> B{Check Redis Connection}

  B -->|Connection Failed| B_0[Start Health Check Scheduler]
  B_0 --> B_0_0[Fallback Mode]

  subgraph "Initialization"
    B -->|Connection Success| B_1{Check Unsynced Files}
    B_1 -->|Exist| B_1_1[Sync Data to Redis]
  end

  subgraph "Normal Mode"

    subgraph "Normal Mode Read"
    C --> D{Query Redis}
    D -->|Found| D_1[Update Memory Cache]
    D -->|Not Found| D_0{Check Memory Cache}

    D_0 -->|Found| D_0_1[Sync to Redis]
    end

    subgraph "Normal Mode Write"
    E --> F{Write to Redis}
    F -->|Success| F_1[Write to Memory]
    F -->|Failed| F_0{Check Redis Connection}

    F_0 -->|Connection Success, Attempts <= 3| E
    end
  end

  D_1 --> ReturnResult[Return Result]
  D_0 -->|Not Found| ReturnResult
  D_0_1 --> ReturnResult

  I_0 -->|Not Found| ReturnResult[Return null]
  I_0_1 --> ReturnResult[Return Result]

  B_1_1 --> B_1_0
  B_1 -->|Not Exist| B_1_0[Normal Mode]
  B_1_0 --> C[Read Request]
  B_1_0 --> E[Write Request]

  F_0 -->|Failed| O
  F_0 --> B_0[Start Health Check Scheduler]

  B_0_0 --> J{Check Redis Connection/Every ? seconds}
  B_0_0 --> N[Write Request]

  subgraph "Fallback Mode"
    subgraph "Fallback Mode Read"
    B_0_0 --> H[Read Request]
    I_0 -->|Found| I_0_1[Update Memory Cache]
    end

    subgraph "Fallback Mode Monitor"
    J -->|Recovered| J_1[Execute Recovery Process]
    J -->|Not Recovered| J_0[Continue Fallback Mode]
    J_0 --> J

    J_1 --> K[Sync Memory Data to Redis]
    K --> L[Sync JSON to Redis]
    L --> M{Sync Status}
    M -->|Failed, Attempts <= 3| J_1
    end

    subgraph "Fallback Mode Write"
    N--> O[Update Memory Cache]
    O --> P{DB Folder Exists}
    P --> |Yes| P_1[Write Individual Files]
    P --> |No| P_0[Create DB Folder]
    P_0 --> P_1
    end
  end

  M -->|Success| B_1_0

  H --> Q{Query Memory Cache}
  S -->|Not Found| I_0{Check JSON Exists}

  subgraph "Memory Flow"
    subgraph "Memory Read"
    Q{Check Expiration} -->|Expired| Q_1[Remove Cache and Delete JSON]
    Q_1 --> |null| S
    Q --> |Not Expired| S[Return Result]
    end 

    subgraph "Memory Cleanup"
    T[Memory Cleanup/Every ? seconds] --> U[Clean Memory Data]
    U --> V[Remove JSON]
    V --> T
    end 
  end
Enter fullscreen mode Exit fullscreen mode

Dependencies

How to use

Installation

go get github.com/pardnchiu/go-redis-fallback
Enter fullscreen mode Exit fullscreen mode

Initialization

package main

import (
  "log"
  "time"

  rf "github.com/pardnchiu/go-redis-fallback"
)

func main() {
  // Minimal configuration
  config := rf.Config{
    Redis: &rf.Redis{
      Host:     "localhost",
      Port:     6379,
      Password: "",
      DB:       0,
    },
  }

  // Initialize Redis fallback mechanism
  client, err := rf.New(config)
  if err != nil {
    log.Fatal(err)
  }
  defer client.Close()

  // Store data (with TTL support)
  err = client.Set("user:1", map[string]string{
    "name":  "John",
    "email": "[email protected]",
  }, 5*time.Minute)

  // Get data
  value, err := client.Get("user:1")
  if err == nil {
    log.Printf("Value: %v", value)
  }

  // Delete data
  err = client.Del("user:1")
}
Enter fullscreen mode Exit fullscreen mode

Configuration Details

type Config struct {
  Redis   *Redis   // Redis configuration (required)
  Log     *Log     // Logging configuration (optional)
  Options *Options // System parameters and fallback settings (optional)
}

type Redis struct {
  Host     string // Redis server host address (required)
  Port     int    // Redis server port number (required)
  Password string // Redis authentication password (optional, empty for no auth)
  DB       int    // Redis database index (required, typically 0-15)
}

type Log struct {
  Path      string // Log directory path (default: ./logs/redisFallback)
  Stdout    bool   // Enable console output logging (default: false)
  MaxSize   int64  // Maximum log file size before rotation in bytes (default: 16MB)
  MaxBackup int    // Number of rotated log files to retain (default: 5)
  Type      string // Output format: "json" for slog standard, "text" for tree format (default: "text")
}

type Options struct {
  DBPath      string        // File storage path (default: ./files/redisFallback/db)
  MaxRetry    int           // Redis retry count (default: 3)
  MaxQueue    int           // Write queue size (default: 1000)
  TimeToWrite time.Duration // Batch write interval (default: 3 seconds)
  TimeToCheck time.Duration // Health check interval (default: 1 minute)
}
Enter fullscreen mode Exit fullscreen mode

Supported Operations

Core Methods

// Store data with optional TTL
err := client.Set("key", value, ttl)

// Retrieve data
value, err := client.Get("key")

// Delete data
err := client.Del("key")

// Close client and cleanup resources
err := client.Close()
Enter fullscreen mode Exit fullscreen mode

Storage Modes

// Normal Mode - Redis available
// 1. Write to Redis first
// 2. Update memory cache on success
// 3. Background sync for consistency

// Fallback Mode - Redis unavailable
// 1. Update memory cache immediately
// 2. Queue write operations
// 3. Batch write to local files
// 4. Monitor Redis health
Enter fullscreen mode Exit fullscreen mode

Core Features

Connection Management

  • New - Create new Redis fallback client
  client, err := rf.New(config)
Enter fullscreen mode Exit fullscreen mode
  • Initialize Redis connection
  • Setup logging system
  • Check for unsynced files
  • Start in appropriate mode

    • Close - Close Redis fallback client
  err := client.Close()
Enter fullscreen mode Exit fullscreen mode
  • Close Redis connection
  • Flush pending writes
  • Release system resources

Storage Features

  • Three-tier Architecture - Memory cache as first layer, Redis as second, local files as fallback
  value, err := client.Get("key") // Checks memory → Redis → local files
Enter fullscreen mode Exit fullscreen mode
  • Automatic Fallback - Seamlessly switch to local storage when Redis fails
  err := client.Set("key", value, ttl) // Falls back to local storage on Redis failure
Enter fullscreen mode Exit fullscreen mode
  • Smart Recovery - Batch synchronize data when Redis becomes available
  // Automatically triggered during health checks
  // Syncs memory and file data back to Redis
Enter fullscreen mode Exit fullscreen mode

Fallback Flow

  • Health Monitoring - Periodic Redis connection checks
  // Automatically runs every TimeToCheck interval
  // Attempts recovery when Redis is available
Enter fullscreen mode Exit fullscreen mode
  • Batch Operations - Optimize performance during fallback
  // Queue writes in memory
  // Batch write to files every TimeToWrite interval
  // Batch sync to Redis during recovery
Enter fullscreen mode Exit fullscreen mode
  • Data Persistence - Layered file storage with MD5 encoding
  // Files stored in nested directories based on key hash
  // JSON format with metadata: key, data, type, timestamp, ttl
Enter fullscreen mode Exit fullscreen mode

Security Features

  • Data Integrity: Consistent data across memory, Redis, and local storage
  • Atomic Operations: Prevent data corruption during mode transitions
  • TTL Management: Automatic expiration handling across all storage layers
  • Layered Structure: MD5-based directory structure prevents filesystem bottlenecks
  • Concurrency Safety: Thread-safe operations with proper locking mechanisms

File Storage Structure

Uses MD5 encoding to implement layered directories:

{DBPath}/db/
├── 0/                   # Redis DB number
│   ├── ab/              # First 2 chars of MD5
│   │   ├── cd/          # 3rd-4th chars of MD5
│   │   │   ├── ef/      # 5th-6th chars of MD5
│   │   │   │   └── abcdef1234567890abcdef1234567890.json
Enter fullscreen mode Exit fullscreen mode

File content format:

{
  "key": "original key value",
  "data": "actual stored data",
  "type": "interface {}",
  "timestamp": 1234567890,
  "ttl": 300
}
Enter fullscreen mode Exit fullscreen mode

Feature Progress

Continuous improvement

  • General Operations

    • [x] Get - Retrieve data
    • [x] Set - Store data
    • [x] Del - Delete key-value
    • [ ] Exists - Check if key exists
    • [ ] Expire/ExpireAt - Set expiration time
    • [ ] TTL - Get remaining time to live
    • [ ] Keys - Find keys matching pattern
    • [ ] Scan - Iterate keys
    • [ ] Pipeline - Batch commands
    • [ ] TxPipeline - Transaction batch
  • String Operations

    • [ ] SetNX - Set if not exists
    • [ ] SetEX - Set with expiration time
    • [ ] Incr/IncrBy - Increment numeric value
    • [ ] Decr/DecrBy - Decrement numeric value
    • [ ] MGet/MSet - Batch get/set multiple key-value pairs
  • Hash Operations

    • [ ] HSet/HGet - Set/get hash field
    • [ ] HGetAll - Get all fields and values
    • [ ] HKeys/HVals - Get all field names/values
    • [ ] HDel - Delete hash field
    • [ ] HExists - Check if field exists
  • List Operations

    • [ ] LPush/RPush - Add elements from left/right
    • [ ] LPop/RPop - Remove elements from left/right
    • [ ] LRange - Get range elements
    • [ ] LLen - Get list length
  • Set Operations

    • [ ] SAdd - Add element to set
    • [ ] SMembers - Get all set members
    • [ ] SRem - Remove element from set
    • [ ] SCard - Get set cardinality
    • [ ] SIsMember - Check if element is in set

Cannot be supported in fallback mode

  • Blocking Operations

    • BLPop/BRPop - Blocking left/right pop
  • Sorted Set Operations

    • ZAdd - Add element to sorted set
    • ZRange/ZRevRange - Get range by score
    • ZRank/ZRevRank - Get element rank
    • ZScore - Get element score
    • ZRem - Remove element
  • Pub/Sub

    • Publish - Publish message
    • Subscribe - Subscribe to channel
  • Lua Scripts

    • Eval/EvalSha - Execute Lua script

License

This source code project is licensed under the MIT License.


©️ 2025 邱敬幃 Pardn Chiu

Top comments (0)