Elsayed Kamal

Posted on May 31

Laravel Queues Explained: Building Scalable Background Job Processing Systems

#laravel #php #webdev #tutorial

Laravel Queues Explained: Building Scalable Background Job Processing Systems

Laravel's queue system is a game-changer for building responsive web applications. Instead of forcing users to wait for time-consuming operations like sending emails, processing images, or generating reports, Laravel queues allow you to defer these tasks to background workers. This comprehensive guide will walk you through everything you need to know about implementing and optimizing Laravel queues.

Why Laravel Queues Matter

The Problem with Synchronous Processing

Imagine a user uploading a large image to your application. Without queues, the user would have to wait while your server:

Validates the image
Resizes it to multiple formats
Optimizes file size
Uploads to cloud storage
Updates the database
Sends notification emails

This could take 30+ seconds, creating a poor user experience.

The Queue Solution

With Laravel queues, you can:

Accept the upload instantly
Return immediate feedback to the user
Process everything in the background
Notify the user when complete

Setting Up Laravel Queues

1. Choose Your Queue Driver

Laravel supports multiple queue backends:

# Database (good for development/small apps)
QUEUE_CONNECTION=database

# Redis (recommended for production)
QUEUE_CONNECTION=redis

# Amazon SQS (for AWS environments)
QUEUE_CONNECTION=sqs

# Sync (for testing - processes immediately)
QUEUE_CONNECTION=sync

2. Database Queue Setup

For database queues, create the necessary tables:

php artisan queue:table
php artisan queue:failed-table
php artisan migrate

3. Redis Queue Setup

Install Redis and configure the connection:

# Install Redis
sudo apt-get install redis-server

# Install PHP Redis extension
composer require predis/predis

Update your .env:

REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

Creating and Dispatching Jobs

Basic Job Creation

Generate a new job class:

php artisan make:job ProcessImageUpload

This creates a job class with the following structure:

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use App\Models\Image;
use Intervention\Image\Facades\Image as ImageProcessor;

class ProcessImageUpload implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $image;

    // Job configuration
    public $timeout = 120; // 2 minutes
    public $tries = 3; // Retry up to 3 times
    public $maxExceptions = 1; // Stop after 1 exception

    public function __construct(Image $image)
    {
        $this->image = $image;
    }

    public function handle()
    {
        // Create thumbnails
        $this->createThumbnails();

        // Optimize original image
        $this->optimizeImage();

        // Upload to cloud storage
        $this->uploadToCloud();

        // Update database
        $this->updateDatabase();

        // Send notification
        $this->notifyUser();
    }

    private function createThumbnails()
    {
        $sizes = [150, 300, 600];

        foreach ($sizes as $size) {
            $thumbnail = ImageProcessor::make($this->image->path)
                ->resize($size, $size, function ($constraint) {
                    $constraint->aspectRatio();
                    $constraint->upsize();
                });

            $thumbnailPath = storage_path("thumbnails/{$size}_{$this->image->filename}");
            $thumbnail->save($thumbnailPath, 85);
        }
    }

    private function optimizeImage()
    {
        $optimized = ImageProcessor::make($this->image->path)
            ->encode('jpg', 90);

        $optimized->save($this->image->path);
    }

    // Additional methods...
}

Dispatching Jobs

There are several ways to dispatch jobs:

// Basic dispatch
ProcessImageUpload::dispatch($image);

// Dispatch with delay
ProcessImageUpload::dispatch($image)
    ->delay(now()->addMinutes(5));

// Dispatch to specific queue
ProcessImageUpload::dispatch($image)
    ->onQueue('image-processing');

// Conditional dispatch
ProcessImageUpload::dispatchIf($image->needs_processing, $image);
ProcessImageUpload::dispatchUnless($image->is_processed, $image);

// Chain jobs
ProcessImageUpload::withChain([
    new OptimizeImage($image),
    new GenerateMetadata($image),
    new SendNotification($image->user)
])->dispatch($image);

Advanced Queue Features

Job Batching

Process multiple related jobs together:

use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;

$images = Image::where('user_id', $userId)->get();

$batch = Bus::batch(
    $images->map(fn($image) => new ProcessImageUpload($image))
)->then(function (Batch $batch) {
    // All images processed successfully
    Log::info("Batch {$batch->id} completed successfully");
})->catch(function (Batch $batch, Throwable $e) {
    // Handle batch failure
    Log::error("Batch {$batch->id} failed: " . $e->getMessage());
})->finally(function (Batch $batch) {
    // Cleanup regardless of success/failure
    Log::info("Batch {$batch->id} finished processing");
})->name('Image Processing Batch')
->allowFailures()
->dispatch();

// Check batch progress
$progress = $batch->processedPercentage();

Job Middleware

Create reusable logic for jobs:

<?php

namespace App\Jobs\Middleware;

class RateLimited
{
    protected $key;
    protected $maxAttempts;
    protected $decayInSeconds;

    public function __construct($key, $maxAttempts = 10, $decayInSeconds = 60)
    {
        $this->key = $key;
        $this->maxAttempts = $maxAttempts;
        $this->decayInSeconds = $decayInSeconds;
    }

    public function handle($job, $next)
    {
        if (RateLimiter::tooManyAttempts($this->key, $this->maxAttempts)) {
            // Release job back to queue with delay
            $job->release(RateLimiter::availableIn($this->key));
            return;
        }

        RateLimiter::hit($this->key, $this->decayInSeconds);

        $next($job);
    }
}

Apply middleware to jobs:

public function middleware()
{
    return [
        new RateLimited("image-processing:{$this->image->user_id}", 5, 60)
    ];
}

Handling Failed Jobs

Implement proper error handling:

public function failed(Exception $exception)
{
    // Log the failure
    Log::error('Image processing failed', [
        'image_id' => $this->image->id,
        'error' => $exception->getMessage(),
        'trace' => $exception->getTraceAsString()
    ]);

    // Update image status
    $this->image->update(['status' => 'failed']);

    // Notify user
    $this->image->user->notify(new ImageProcessingFailed($this->image));

    // Send admin notification for critical failures
    if ($exception instanceof CriticalException) {
        Notification::route('slack', config('slack.admin_webhook'))
            ->notify(new CriticalJobFailure($this));
    }
}

Running Queue Workers

Development

For development, use the simple worker:

# Process default queue
php artisan queue:work

# Process specific queues with priority
php artisan queue:work --queue=high,default,low

# Stop after processing 100 jobs (prevents memory leaks)
php artisan queue:work --max-jobs=100

# Stop after 1 hour
php artisan queue:work --max-time=3600

Production with Supervisor

Create a Supervisor configuration:

[program:laravel-queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/your-app/artisan queue:work redis --sleep=3 --tries=3 --max-time=3600 --max-jobs=1000
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/your-app/storage/logs/queue-worker.log
stdout_logfile_maxbytes=100MB
stdout_logfile_backups=5

Start the workers:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start laravel-queue-worker:*

Queue Monitoring and Optimization

Laravel Horizon

For Redis queues, use Laravel Horizon for monitoring:

composer require laravel/horizon
php artisan horizon:install
php artisan migrate

Configure queues in config/horizon.php:

'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['high', 'default', 'low'],
            'balance' => 'auto',
            'maxProcesses' => 10,
            'maxTime' => 0,
            'maxJobs' => 0,
            'memory' => 512,
            'tries' => 3,
            'timeout' => 60,
            'nice' => 0,
        ],
    ],
],

Start Horizon:

php artisan horizon

Performance Monitoring

Track queue performance with custom metrics:

// In your job's handle method
$startTime = microtime(true);

// Your job logic here...

$duration = microtime(true) - $startTime;

// Log performance metrics
Log::info('Job performance', [
    'job' => self::class,
    'duration' => $duration,
    'memory_peak' => memory_get_peak_usage(true),
    'queue' => $this->queue,
]);

// Report to monitoring service
app('metrics')->timing('queue.job.duration', $duration, [
    'job' => self::class,
    'queue' => $this->queue,
]);

Real-World Use Cases

Email Campaign Processing

class SendEmailCampaign implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $campaign;
    public $timeout = 300; // 5 minutes
    public $tries = 2;

    public function __construct(EmailCampaign $campaign)
    {
        $this->campaign = $campaign;
    }

    public function handle()
    {
        $recipients = $this->campaign->getRecipients();

        // Batch recipients to avoid overwhelming the mail service
        $batches = $recipients->chunk(100);

        foreach ($batches as $batch) {
            dispatch(new SendEmailBatch($this->campaign, $batch))
                ->onQueue('email-sending')
                ->delay(now()->addSeconds(10)); // Rate limiting
        }

        $this->campaign->update(['status' => 'sending']);
    }
}

Data Export Processing

class GenerateUserReport implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $reportRequest;
    public $timeout = 1800; // 30 minutes for large reports

    public function handle()
    {
        $filename = "user_report_{$this->reportRequest->id}.xlsx";
        $filePath = storage_path("reports/{$filename}");

        // Generate Excel file
        Excel::store(
            new UsersExport($this->reportRequest->filters),
            $filename,
            'reports'
        );

        // Upload to S3
        $s3Path = Storage::disk('s3')->putFile('reports', $filePath);

        // Update request with download link
        $this->reportRequest->update([
            'status' => 'completed',
            'download_url' => Storage::disk('s3')->url($s3Path),
            'expires_at' => now()->addDays(7)
        ]);

        // Notify user
        $this->reportRequest->user->notify(
            new ReportReady($this->reportRequest)
        );

        // Cleanup local file
        unlink($filePath);
    }
}

Best Practices and Tips

1. Job Design

Keep jobs focused on a single responsibility
Make jobs idempotent (safe to run multiple times)
Use type hints for better error reporting
Implement proper timeout and retry logic

2. Queue Management

Use different queues for different priorities
Monitor queue sizes and processing times
Implement proper logging and alerting
Use Redis for high-throughput applications

3. Error Handling

Always implement the failed() method
Log detailed error information
Implement graceful degradation
Set up alerts for critical failures

4. Performance Optimization

Use job batching for related tasks
Implement rate limiting for external APIs
Monitor memory usage and restart workers
Use appropriate timeout values

5. Security Considerations

Validate job data thoroughly
Use queue encryption for sensitive data
Implement proper access controls
Monitor for queue injection attacks

Testing Queue Jobs

// In your test file
use Illuminate\Support\Facades\Queue;

class ImageUploadTest extends TestCase
{
    public function test_image_upload_dispatches_processing_job()
    {
        Queue::fake();

        $user = User::factory()->create();
        $this->actingAs($user);

        $file = UploadedFile::fake()->image('test.jpg');

        $response = $this->post('/upload', ['image' => $file]);

        Queue::assertPushed(ProcessImageUpload::class, function ($job) use ($file) {
            return $job->image->filename === $file->getClientOriginalName();
        });
    }

    public function test_job_processes_image_correctly()
    {
        $image = Image::factory()->create();

        $job = new ProcessImageUpload($image);
        $job->handle();

        $this->assertTrue($image->fresh()->is_processed);
        $this->assertFileExists(storage_path("thumbnails/150_{$image->filename}"));
    }
}

Conclusion

Laravel queues are essential for building scalable, responsive applications. They allow you to:

Improve user experience by deferring slow operations
Scale your application horizontally
Handle failures gracefully
Process large volumes of background tasks

Start with database queues for development, then move to Redis for production. Always implement proper monitoring, error handling, and testing to ensure your queue system is robust and reliable.

Remember: A well-designed queue system is like a good foundation – invisible when working properly, but essential for supporting everything built on top of it.

Have you implemented Laravel queues in your projects? Share your experiences and challenges in the comments below! What queue drivers do you prefer and why?

DEV Community

Laravel Queues Explained: Building Scalable Background Job Processing Systems

Laravel Queues Explained: Building Scalable Background Job Processing Systems

Why Laravel Queues Matter

The Problem with Synchronous Processing

The Queue Solution

Setting Up Laravel Queues

1. Choose Your Queue Driver

2. Database Queue Setup

3. Redis Queue Setup

Creating and Dispatching Jobs

Basic Job Creation

Dispatching Jobs

Advanced Queue Features

Job Batching

Job Middleware

Handling Failed Jobs

Running Queue Workers

Development

Production with Supervisor

Queue Monitoring and Optimization

Laravel Horizon

Performance Monitoring

Real-World Use Cases

Email Campaign Processing

Data Export Processing

Best Practices and Tips

1. Job Design

2. Queue Management

3. Error Handling

4. Performance Optimization

5. Security Considerations

Testing Queue Jobs

Conclusion

Top comments (0)