The Async Performance Trap We Fell Into
Our database migration script was taking 8 hours to process 50,000 records. The code looked elegant:
await Promise.all(records.map(processRecord)); // "This should be fast!"
But it crashed our database with 500 concurrent connections.
Here’s how we fixed it—and when loops actually beat Promise.all.
1. The Dark Side of Promise.all
🚨 Problem #1: Uncontrolled Concurrency
// Spawns 50,000 promises at once!
await Promise.all(hugeArray.map(async (item) => {
await db.insert(item); // 💥 Database melts down
}));
✅ Fix: Batch with p-limit
import pLimit from 'p-limit';
const limit = pLimit(10); // Max 10 concurrent ops
await Promise.all(
hugeArray.map(item => limit(() => db.insert(item)))
);
🚨 Problem #2: No Error Handling
// One failure rejects ALL promises
await Promise.all([
fetchUser(1),
fetchUser(2), // If this fails, entire batch dies
]);
✅ Fix: Use Promise.allSettled + filtering
const results = await Promise.allSettled(promises);
const successes = results.filter(r => r.status === 'fulfilled');
🚨 Problem #3: Memory Overload
// Loads ALL records into memory before processing
const records = await getAllRecords(); // 500MB array
await Promise.all(records.map(processRecord));
✅ Fix: Stream with for-await-of
for await (const record of getRecordsStream()) {
await processRecord(record); // Processes one at a time
}
2. When Old-School Loops Win
| Scenario | Promise.all |
Loop |
|---|---|---|
| Order matters | ❌ No | ✅ Yes |
| Memory-constrained | ❌ No | ✅ Yes |
| Backpressure needed | ❌ No | ✅ Yes |
Example: Sequential Processing
// Slower but safer
for (const item of items) {
await process(item); // One at a time
}
Example: Batched Loops
// Best of both worlds
const batchSize = 100;
for (let i = 0; i < items.length; i += batchSize) {
const batch = items.slice(i, i + batchSize);
await Promise.all(batch.map(process)); // Controlled concurrency
}
3. Real-World Benchmarks
| Method | Time (10K ops) | Memory Usage |
|---|---|---|
Promise.all (unlimited) |
2 min 💥 | 1.2GB 🚨 |
p-limit (concurrency 10) |
8 min | 200MB |
| Batched loops (100/chunk) | 6 min | 50MB |
Lesson learned: "Faster" isn’t always better.
Key Takeaways
⚡ Promise.all is great for → Small, independent, non-IO-heavy tasks
🐢 Loops are better for → Order-sensitive, memory-bound, or backpressured workflows
🛠 Hybrid approach → Batched Promise.all inside loops
Which async pattern burned you? Share your war story!
Top comments (0)