Yi Wan

Posted on Jun 6

Cloudflare Worker Memory Overflow Analysis: How 20MB File Crashed 128MB Runtime

TL;DR

Encountered Worker exceeded memory limit when processing 20MB files in Cloudflare Worker. Root cause: Base64 encoding + double memory allocation. Solution: Let Resend fetch files directly from R2 URLs instead of relaying through Worker.

Problem Context

Building email delivery for Zin Flow (web-to-EPUB converter) using:

Cloudflare Worker (128MB memory limit)
Cloudflare R2 (object storage)
Resend (email service)

🚨 Issue Reproduction

Error Message

Worker exceeded memory limit.

Problematic Code

// Fetch file from R2
const file = await env.R2_BUCKET.get(r2Key);

if (!file) {
  console.error(`File not found in R2: ${r2Key}`);
  return;
}

try {
  // ❌ Problem 1: First memory copy
  const fileData = await file.arrayBuffer(); // 20MB

  // ❌ Problem 2: Base64 encoding creates second copy
  const fileBase64 = arrayBufferToBase64(fileData); // ~27MB

  // Send email with attachment
  await resend.emails.send({
    attachments: [{
      filename: 'book.epub',
      content: fileBase64
    }]
  });
} catch (error) {
  console.error('Memory error:', error);
}

🔍 Deep Analysis

1. Memory Usage Breakdown

Component	Size	Notes
Original ArrayBuffer	20MB	Raw file size
Base64 String	~27MB	Base64 adds 33% overhead
V8 Engine Overhead	~30MB	Runtime base consumption
Object Metadata	~5MB	String headers, ArrayBuffer metadata
Total	~82MB	Approaching 128MB limit

2. Base64 Encoding Hidden Costs

function arrayBufferToBase64(buffer) {
  // Creates Uint8Array view (additional memory overhead)
  const bytes = new Uint8Array(buffer);

  // String concatenation creates temporary memory
  let binary = '';
  for (let i = 0; i < bytes.byteLength; i++) {
    binary += String.fromCharCode(bytes[i]);
  }

  // btoa() internal implementation needs extra memory
  return btoa(binary);
}

3. Edge Runtime Memory Characteristics

In Cloudflare's Edge Runtime, V8 memory management has specific traits:

Delayed GC: No immediate garbage collection under low pressure
String Deduplication: Large strings typically aren't deduplicated
ArrayBuffer Handling: May use off-heap memory but still counts toward limit

💡 Solution Evolution

Approach 1: Memory Optimization (Failed)

// Attempted manual memory release
const fileData = await file.arrayBuffer();
const fileBase64 = arrayBufferToBase64(fileData);

// ❌ Setting null doesn't immediately free memory
fileData = null;

// ❌ Manual GC unavailable in Workers
// global.gc?.();

Approach 2: Streaming Processing (API Limited)

// ❌ Resend doesn't support streaming uploads
const stream = file.body;
// Still need to convert to complete Base64 string

Approach 3: Architectural Redesign (✅ Final Solution)

Key insight: Resend can fetch files directly from URLs!

export default {
  async fetch(request, env) {
    const { r2Key, userEmail, fileName } = await request.json();

    // ✅ Generate public R2 URL instead of processing file
    const fileUrl = `https://${env.R2_DOMAIN}/${r2Key}`;

    // ✅ Let Resend download directly from R2
    const emailResult = await resend.emails.send({
      to: userEmail,
      subject: 'Your Zin Flow EPUB is Ready',
      html: createEmailTemplate(fileName),
      attachments: [{
        filename: fileName,
        path: fileUrl // Resend handles the download!
      }]
    });

    return new Response(JSON.stringify({
      success: true,
      messageId: emailResult.id
    }));
  }
};

function createEmailTemplate(fileName) {
  return `
    <div style="font-family: system-ui, sans-serif; max-width: 600px;">
      <h2>📚 Your EPUB is Ready!</h2>
      <p>Your ebook "${fileName}" has been converted and attached to this email.</p>

      <div style="background: #f8f9fa; padding: 20px; border-radius: 8px; margin: 20px 0;">
        <h3>📧 For Kindle Users:</h3>
        <ol>
          <li>Forward this email to your Kindle email address</li>
          <li>Or download the attachment and transfer via USB</li>
        </ol>
      </div>

      <p><small>Powered by <a href="https://zinflow.app">Zin Flow</a> - Available on 
      <a href="https://apps.apple.com/app/zin-flow/id6670145951">App Store</a></small></p>
    </div>
  `;
}

📊 Performance Comparison

Old vs New Architecture

Metric	Old Architecture	New Architecture	Improvement
Memory Peak	~82MB	~2MB	97%↓
Response Time	3-8 seconds	50-150ms	98%↓
Error Rate	15%*	<0.1%	99%↓
File Size Limit	20MB (soft)	Unlimited**	∞
User Experience	Wait + Potential Failure	Instant + Reliable	Qualitative Leap

*For large files

**Limited by email provider attachment limits

🛠️ Best Practices

1. Cloudflare Worker Memory Guidelines

// ✅ Good: Lightweight data processing
async function processMetadata(data) {
  const result = await lightweightOperation(data);
  return result;
}

// ❌ Avoid: Large file manipulation
async function processLargeFile(file) {
  const buffer = await file.arrayBuffer(); // High memory usage
  const processed = heavyProcessing(buffer); // Double allocation
  return processed;
}

2. Memory Monitoring Techniques

// Monitor memory usage in Workers
function logMemoryUsage(label) {
  const memory = performance.memory;
  if (memory) {
    console.log(`${label}:`, {
      used: `${(memory.usedJSHeapSize / 1024 / 1024).toFixed(2)}MB`,
      total: `${(memory.totalJSHeapSize / 1024 / 1024).toFixed(2)}MB`,
      limit: `${(memory.jsHeapSizeLimit / 1024 / 1024).toFixed(2)}MB`
    });
  }
}

logMemoryUsage('Before operation');
// ... perform operation
logMemoryUsage('After operation');

3. Alternative Architecture Patterns

// Pattern 1: Direct URL passing
const publicUrl = generatePublicUrl(fileKey);
await emailService.send({ attachments: [{ path: publicUrl }] });

// Pattern 2: Signed URLs with expiration
const signedUrl = await bucket.sign(key, { expiresIn: 3600 });

// Pattern 3: Redirect response
return new Response(null, {
  status: 302,
  headers: { 'Location': fileUrl }
});

// Pattern 4: Chunked processing
async function processInChunks(file, chunkSize = 1024 * 1024) {
  const stream = file.body;
  const reader = stream.getReader();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    // Process chunk without loading entire file
    await processChunk(value);
  }
}

🎯 Key Learnings

Edge Runtime Reality: 128MB ≠ 128MB usable memory
Base64 Tax: Encoding adds 33% size + processing overhead
Architecture First: Sometimes best optimization is avoiding the problem
Service Integration: Leverage external service capabilities (Resend's URL fetching)

🚀 Real-World Impact

This solution powers Zin Flow's email delivery system, serving thousands of EPUB conversions. The app is available on App Store for iOS and Mac users who want to convert web articles into beautiful ebooks.

📚 Further Reading

🏷️ Tags

#cloudflare #workers #javascript #memory-optimization #architecture #performance #resend #email

💡 Pro Tip: When designing edge functions, always consider the hidden costs of data transformations. Sometimes the most elegant solution is letting specialized services handle what they do best.

Discussion: Have you encountered similar memory issues with Cloudflare Workers? Share your solutions in the comments! Also, check out Zin Flow if you're interested in web-to-EPUB conversion tools.

DEV Community