Overview
We’ll explore two important concepts:
🔄 Implementing Caching – We will use Redis Cloud, a managed Redis service, to temporarily store the results of API calls. This will simulate caching by allowing us to save and retrieve responses externally. When the same request is made again, we will quickly return the stored data instead of calling the external API again. This will reduce response time and improve performance.
🛡️ Protecting the API with IP Rate Limiting – To prevent abuse, we will implement a basic rate limiter that counts how many times a user (based on IP address) accesses the API across all its routes. If the number of requests exceeds the allowed limit within a certain time window, that IP will be temporarily blocked. In our example, users will be limited to 20 requests per hour. Once the time window resets, the user will be able to access the API again.
What is Redis? 🔍
Redis is an open-source, in-memory data structure store commonly used as a cache or message broker. It supports various data types, such as strings, hashes, lists, sets, and more. Redis is known for its high performance and low latency, making it ideal for scenarios where fast data access is required. It's often used to store temporary data, such as session information, frequently accessed API responses, and real-time analytics, reducing the load on databases and improving overall application performance.
How does Redis work? ⚙️
Redis operates as an in-memory data store, meaning it keeps all of its data in RAM rather than on disk, enabling sub-millisecond data access times. Redis supports different data structures like strings, hashes, lists, sets, sorted sets, and more. It works through key-value pairs where each key is associated with a value.
Data in Redis is stored persistently (optional) through mechanisms like snapshots (RDB) or append-only files (AOF) to disk, allowing it to recover data in case of failures. Redis uses an event-driven architecture and provides high availability and horizontal scaling options with features like replication and sharding, making it ideal for use cases that require low-latency data access and high-throughput performance, such as caching, session storage, and real-time analytics.
Step 0 🚀
In this tutorial, instead of installing Redis locally, we'll be using Redis Enterprise Cloud to create a Redis database. Redis Enterprise Cloud offers a managed service, allowing us to set up a Redis instance in the cloud without the need for local installation. We can easily get started with a free account.
As data source, we’ll use requests to the endpoint https://rickandmortyapi.com/api
We create an empty Node.js project and install the following dependencies:
pnpm add axios dotenv express redis
In the .env file, we add our Redis connection details:
REDIS_HOSTNAME =
REDIS_PORT =
REDIS_PASSWORD =
API_URL = https://rickandmortyapi.com/api/
Step 1 🛠️
We start by importing the necessary dependencies, defining constants like the port, rate limits, and cache expiration, and initializing both the Express app and the Redis client using environment variables.
const express = require('express');
const axios = require('axios');
const { createClient } = require('redis');
require('dotenv').config();
// App configuration
const PORT = process.env.PORT || 4000;
const RATE_LIMIT = 20;
const RATE_LIMIT_WINDOW = 3600; // 1 hour in seconds
const CACHE_EXPIRATION = 20; // Cache for 20 seconds
const app = express();
// Initialize Redis client using environment variables
const client = createClient({
password: process.env.REDIS_PASSWORD,
socket: {
host: process.env.REDIS_HOSTNAME,
port: process.env.REDIS_PORT
}
});
// Log Redis errors
client.on('error', (err) => {
console.error('Redis connection error:', err);
});
Step 2 🧱
Next, we add middleware for rate limiting and define a route to get character data. We use Redis to cache responses and return cached results when available.
// Apply rate limiter to all routes
app.use(limitRate);
// API route to fetch character data (cached using Redis)
app.get('/api/character', getCachedData);
/**
* Redis caching logic
* If data is found in Redis, return it.
* Otherwise, fetch it from the external API and cache it.
*/
async function getCachedData(req, res) {
try {
const keyParam = 'character';
// Check if data exists in Redis cache
const cachedData = await client.get(keyParam);
if (cachedData) {
console.log(`Cache hit for ${keyParam}`);
return res.json(JSON.parse(cachedData));
}
// If not cached, fetch data from external API
console.log(`Cache miss for ${keyParam}, fetching from API`);
const apiUrl = `${process.env.API_URL}${keyParam}`;
const response = await axios.get(apiUrl);
// Store the result in Redis with an expiration time
await client.set(keyParam, JSON.stringify(response.data), {
EX: CACHE_EXPIRATION
});
return res.json(response.data);
} catch (error) {
console.error('Error fetching data:', error);
const statusCode = error.response?.status || 500;
return res.status(statusCode).json({
error: error.message,
statusCode
});
}
}
/**
* Middleware to limit the number of requests from each IP
* Uses Redis to track request count within a time window
*/
async function limitRate(req, res, next) {
try {
// Get IP address
const ip =
req.headers['cf-connecting-ip'] ||
(req.headers['x-forwarded-for'] || req.socket.remoteAddress || '')
.split(',')[0]
.trim();
const rateKey = `rate:${ip}`;
// Increment the request count for this IP
const requests = await client.incr(rateKey);
// Set expiration if this is the first request
if (requests === 1) {
await client.expire(rateKey, RATE_LIMIT_WINDOW);
}
console.log(`IP ${ip} - Request count: ${requests}`);
// Block the request if the limit is exceeded
if (requests > RATE_LIMIT) {
return res.status(429).json({
status: 'error',
requestCount: requests,
message: 'Rate limit exceeded. Please try again later.'
});
}
// Allow request
next();
} catch (error) {
console.error('Rate limiting error:', error);
next(); // Fail-open: allow request if Redis fails
}
}
Step 3 🔐
Finally, we connect to Redis, start the server, and handle graceful shutdown on termination.
/**
* Connect to Redis and start the Express server
*/
async function startServer() {
try {
await client.connect();
console.log('Connected to Redis successfully');
app.listen(PORT, () => {
console.log(`Server running at http://localhost:${PORT}`);
});
} catch (error) {
console.error('Failed to start server:', error);
process.exit(1);
}
}
/**
* Shutdown Redis connection
*/
process.on('SIGINT', async () => {
try {
await client.quit();
console.log('Redis connection closed');
process.exit(0);
} catch (err) {
console.error('Error shutting down:', err);
process.exit(1);
}
});
// Start server
startServer();
To provide a practical explanation, I’ve deployed our project to Render.
To avoid headaches, remember to upload the project's environment variables in the Settings section of Render and specify node src/index.js in the Start field.
First let's go to our endpoint set up to fetch all characters from the Rick and Morty API. We can see that it returns the corresponding data.
In our Redis database, the first part corresponds to the information saved after the first request to the API, with the key named "Character" (1). We can also see its corresponding TTL (Time to live) (2), originally set to 20 seconds, and finally the data (3). It is important to note that after 20 seconds, the key will automatically be deleted.
The second key/value corresponds to the IP from which we are making the request (1). We can also see its corresponding TTL (2) and the counter (3) indicating how many times we've accessed the endpoint—just once in this case.
Thanks to the console logs, we can more clearly see this situation in the Render terminal.
A while later, after several requests, we can see that the counter is at 6 and the data is coming from the API, since our request happened more than 20 seconds after the previous one. However, in requests 7 and 8, made shortly after, we can observe how the data is retrieved from Redis. On request 9, a minute later, we see it fetch the data from the external API once again.
As a final test, we perform more than 20 requests to see how our API blocks the IP from which we are making the calls.
We can also observe this from the terminal, where requests 21 and 22 did not hit the external API or our Redis database. Remember that we configured this block for 12 hours—once that time has passed, the IP key/value pair will be deleted, and the counter will reset to 1.
Top comments (0)