Posted inProgramming

Implementing Idempotency in Node.js REST APIs with Redis

Programming tutorial - IT technology blog
Programming tutorial - IT technology blog

Stop Duplicate Actions Before They Hit Your Database

A customer clicks “Pay Now.” The spinner hangs for three seconds. Panicked, they smash the button four more times. Without a safeguard, your server might process that transaction repeatedly, draining their bank account for a single order. This is where idempotency becomes your most important backend feature.

An idempotent operation ensures that performing the same action multiple times produces the same result. In REST APIs, this means if a client retries a request because of a timeout, the server recognizes it, processes it exactly once, and returns the original successful response for every subsequent attempt.

The Cost of Getting It Wrong

I’ve seen systems without this protection charge a user $480 for a $120 subscription because of a brief microservice hiccup. Resolving that single mistake required six hours of manual database patching and a formal apology from the support lead. Mastering idempotency isn’t just a technical preference; it’s about protecting your data integrity and your user’s trust.

A Minimal Implementation

You only need Node.js and a Redis instance to get started. Redis is the industry standard here because it handles key expiration automatically and operates with sub-millisecond latency.

npm install express ioredis

The following example uses ioredis to check for a unique key before executing any business logic:

const express = require('express');
const Redis = require('ioredis');
const redis = new Redis();
const app = express();
app.use(express.json());

app.post('/payments', async (req, res) => {
  const idempotencyKey = req.headers['idempotency-key'];

  if (!idempotencyKey) {
    return res.status(400).json({ error: 'Idempotency-Key header is required' });
  }

  // Look for a cached response first
  const cachedResponse = await redis.get(`idempotency:${idempotencyKey}`);
  if (cachedResponse) {
    return res.status(200).json(JSON.parse(cachedResponse));
  }

  // Process the actual logic (e.g., charge the card)
  const result = { success: true, txnId: 'TXN-9982', total: req.body.amount };
  
  // Cache the result for 24 hours (86400 seconds)
  await redis.set(`idempotency:${idempotencyKey}`, JSON.stringify(result), 'EX', 86400);

  res.status(201).json(result);
});

app.listen(3000, () => console.log('Server live on port 3000'));

The Mechanics of a Robust Request Lifecycle

While the basic check-and-set approach works for personal projects, high-traffic production environments require more resilience. We rely on a specific contract between the client and the server: the Idempotency-Key.

The Client’s Responsibility

The frontend or mobile app must generate a unique V4 UUID for every new transaction. If the network fails and the client retries, it sends the exact same UUID. The server uses this string as the primary lookup key in Redis.

How the Server Handles the Request

  1. Extraction: Pull the Idempotency-Key from the request headers.
  2. Validation: Check Redis immediately. If a result exists, skip the logic and return the cached data.
  3. Locking: Mark the key as “In Progress” to prevent race conditions.
  4. Execution: Perform the heavy lifting, such as calling Stripe or updating your SQL tables.
  5. Persistence: Save the final JSON result in Redis.
  6. Cleanup: Set a TTL (Time to Live) so Redis memory stays lean.

I recommend implementing this as middleware. This approach keeps your controllers focused on business logic while protecting every POST or PUT route with a single line of code.

Defeating Race Conditions with Atomic Operations

Standard logic has a hidden flaw. If two identical requests hit your API within 5ms of each other, both might see an empty Redis cache and trigger double-billing. This is a classic race condition.

Using Redis as a Distributed Lock

We solve this using the Redis SET command with the NX (Set if Not eXists) flag. This allows us to atomically lock the key the moment the first request arrives.

async function idempotencyMiddleware(req, res, next) {
  const key = req.headers['idempotency-key'];
  if (!key) return next();

  const redisKey = `idempotency:${key}`;

  // Try to acquire a lock for 60 seconds
  const lockAcquired = await redis.set(redisKey, 'STARTED', 'NX', 'EX', 60);

  if (!lockAcquired) {
    const status = await redis.get(redisKey);
    if (status === 'STARTED') {
      return res.status(409).json({ error: 'Processing in progress. Please wait.' });
    }
    return res.status(200).json(JSON.parse(status));
  }

  // Wrap res.json to cache the final output automatically
  const originalJson = res.json;
  res.json = (body) => {
    redis.set(redisKey, JSON.stringify(body), 'EX', 86400);
    return originalJson.call(res, body);
  };

  next();
}

Handling Concurrent Retries

If a retry arrives while the first request is still being processed, we return a 409 Conflict. This signals the client to pause before trying again. It is a much cleaner solution than blindly running the logic twice and hoping for the best.

What if your server crashes during execution? The 60-second TTL on the ‘STARTED’ state ensures the lock eventually expires. This allows the client to retry and succeed even after a system failure.

Strategic Deployment Tips

Code is only half the battle. You must follow established conventions to ensure your API remains predictable for other developers.

When to Use Idempotency

Focus your energy on non-idempotent methods. GET requests are idempotent by design; refreshing a profile page 100 times should never change a user’s name. Focus on POST requests where side effects occur, such as creating orders, triggering refunds, or sending batch emails.

Choosing Your TTL

How long should you keep these records? For financial transactions, 24 to 48 hours is the industry sweet spot. This window is wide enough for a client to recover from a bad connection but short enough to keep your Redis instance from bloating. If you need to track duplicates for months, migrate the records from Redis to a permanent store like PostgreSQL or MongoDB after the initial 24 hours.

Monitoring is vital. Always log idempotency “hits.” If you notice a sudden spike in duplicate keys, it usually indicates a bug in the client’s retry logic or a serious latency issue in your infrastructure. These logs act as a canary in the coal mine for your system’s overall health.

Building these safeguards takes extra effort upfront. However, it is the difference between a fragile prototype and a professional financial system. Your users—and your sleep schedule—will thank you.

Share:
Tags: