./ahmedhashim

Jitterbug

I found myself debugging a production API that was spitting out bursts of 500s. Traffic was consistent throughout the day, but the errors came in waves at regular intervals.

This service would just fetch from the database and store the response in Redis, with the URL as the cache key and a set TTL. Many clients were hitting our API at similar times, so it would create a bunch of keys that all expired together.

So when those keys expired, we’d get a thundering herd. We had so many cache misses that we had to backfill them through our webserver all at once, which exhausted our DB connections. Any request that tried to grab a connection from the pool would time out and return a 500.

We could have increased our DB connections, but that seemed like a hack since we knew the bursts were predictable. So we added some jitter to our cache keys’ TTL, which spread the expirations out and solved our problem right away.

I wanted to know more about why jitter helped, so I made Jitterbug. It’s a reproducible load test that runs static and jittered TTLs side by side under heavy traffic and shows you both in Grafana.

Setup

I have 2 endpoints that use Redis:

  1. /static/data?id=N
  2. /jitter/data?id=N

They both cache the full response by URL, meaning you’ll have a key per id. The only difference is the TTL. Static is set to a 10s TTL, and jitter will pick a number between 10 and 15s:

from typing import Literal
import random

Mode = Literal["static", "jitter"]

STATIC_TTL = 10  # seconds
MAX_JITTER = 5   # seconds


def _ttl_ms(mode: Mode) -> int:
    if mode == "static":
        return STATIC_TTL * 1000
    jittered_ttl = STATIC_TTL + random.uniform(0, MAX_JITTER)
    return int(jittered_ttl * 1000)

The IDs only go from 0..999, so each endpoint will have 1000 possible keys. On a cache miss, our app will pretend to do some work for 20-50ms and then write to cache.

Stack

Everything runs on Docker here:

  • FastAPI: web server for the /static/data and /jitter/data endpoints
  • Redis: response cache
  • Locust: load generator that drives traffic at a fixed target rate
  • VictoriaMetrics: metrics backend that scrapes /metrics every second
  • Grafana: dashboard used to visualize miss rates and latency
  graph LR
    L["<strong>Locust</strong><br/><span class='mermaid-detail'>1500 users, 4 processes</span>"]
    W["<strong>FastAPI</strong><br/><span class='mermaid-detail'>/static/data?id=N<br/>/jitter/data?id=N</span>"]
    R[("<strong>Redis</strong><br/><span class='mermaid-detail'>Cache</span>")]
    VM["<strong>VictoriaMetrics</strong><br/><span class='mermaid-detail'>scrape every 1s</span>"]
    G["<strong>Grafana</strong>"]

    L -->|"HTTP load"| W
    W -->|"cache read/write by full URL key"| R
    W -->|"/metrics"| VM
    G -->|"PromQL queries"| VM

Traffic was evenly distributed between the static and jitter endpoints. I used Locust with 1500 users and 4 processes to generate ~10.5k req/s combined, and each endpoint was getting ~5.2k req/s on average.

Our app reports cache hits/misses to Prometheus, as well as a latency histogram. We label everything with the mode, and the scrape interval was 1s to catch the short TTLs.

Dashboard

dashboard

The first row is our cache misses/s per endpoint. You can see static spiking periodically, while jitter stays within a narrow range.

The second row is p50, p95 and p99 latency. You can see static’s latency move in tandem with its spikes in misses, whereas jitter stays more even.

The third row combines both. Misses per second shows the raw difference, and the miss ratio panel takes total requests into account.

cache-miss-rate

Results

Here’s a 10 min run on my local box. It handled roughly 6.25 million requests at about 10.4k req/s with no failures.

Latency was good for this workload, with p50 ~16ms, p95 ~50ms and p99 ~74ms.

Static saw about 73k misses and jitter about 59k, giving us a miss ratio of 2.35% vs 1.88%. Most of this is from jitter having a higher average TTL of 12.5 seconds vs 10, which you could also get by increasing static’s TTL. What jitter actually changes is when those misses occur. You can see static jump from close to 0 to 200-275 misses per second, while jitter hovered around 100 for the most part. Jitter’s peak miss rate was about half of static’s.

Why it works

By using a static TTL, you’re going to have keys being born and dying at similar times, causing spikes of misses. When using jitter, you’re spreading those misses out, allowing your backend to handle them more evenly.

You size your database for the worst-case scenario, not the average. By using jitter, you can decrease your worst-case scenario.

Takeaways

Adding jitter to your TTL on frequently accessed API calls is an easy win. It’s a one-line change, and you’ll start seeing results as soon as your traffic spikes.

Jitter doesn’t help when multiple requests are trying to access the same key while it’s being repopulated. We saw about 20% more misses on both endpoints than their TTLs alone would predict, because requests kept hitting keys mid-refill. If you care about this, add some sort of lock or request coalescing in front of the refill.

Feel free to clone the repo and spin up the stack. You’ll notice the difference in the miss and latency panels almost immediately.