./ahmedhashim

How long has it been since an outage forced you into a service dashboard that made everything more confusing? It’s 40 panels deep, every one of them added during some incident at some point, and none of them answer the questions being fired around the incident channel. A director wants to know how much this outage is costing. Support wants to know what customers are experiencing. The on-call engineer wants to know why the pods keep crashing.

I build service dashboards to answer those questions from the top down. At the top, you can see what the service means to the business. In the middle, you can see what users are experiencing. At the bottom, you can see how the machines are doing. Each section targets a different person, so everyone can stop scrolling when they get their answer.

For instance, consider the payments service that powers a typical retail shopping cart. It takes the cart, charges the card, and records the order. Each section below is built for that service, but the same layout could apply to anything you operate.

  flowchart TD
    B["<strong>Business impact</strong><br/><span class='mermaid-detail'>checkouts/min, payment success, revenue</span>"]
    U["<strong>User impact</strong><br/><span class='mermaid-detail'>latency, errors, traffic, saturation</span>"]
    H["<strong>Service health</strong><br/><span class='mermaid-detail'>CPU, memory, restarts, logs</span>"]

    B -->|"explained by"| U
    U -->|"explained by"| H

Here’s the entire dashboard, in Grafana, for the payments service:

Business impact

The top of the dashboard should quantify how valuable the service is to the business. For the payments service, that translates to successful checkouts per minute and revenue generated. Payment success rate also belongs up top, because a run of declined cards is a business problem long before it becomes an engineering problem. A search service would have something different, perhaps the percentage of searches that result in an add to cart. Regardless of your service, the first row should measure your service’s output in units that the business already cares about.

One number doesn’t mean much, so these panels need some sort of baseline. Is 240 checkouts per minute bad? It depends on whether last Tuesday at this time did 250 or 400. Offset the same series by a week and suddenly every panel is a comparison:

sum(rate(checkouts_completed_total[5m])) * 60
sum(rate(checkouts_completed_total[5m] offset 1w)) * 60

If the current line is below last week’s dashed line, nobody debates severity, and this row becomes the screenshot that gets posted in the incident channel. This is also true in reverse. A latency spike that doesn’t show up in this row is most likely something the on-call can fix without paging half the company.

User impact

The middle section reflects what users are experiencing. The four golden signals describe it nicely: latency, traffic, errors, and saturation.

When you’re monitoring latency, think percentiles, not averages. The average hides the slow tail: the payments service might average 40ms, but one in 100 shoppers will see a spinner after clicking pay. I track p50, p95 and p99 on the same chart so you can see the tail alongside the norm:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Each error represents one shopper encountering a failed payment screen. For that reason, the panel should show errors as a ratio, not a raw count. 12 errors per minute sounds bad until you realize your traffic has doubled and you’re still at the same 1% failure rate as before:

sum(rate(http_requests_total{code=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

Traffic gets a panel because the other three signals don’t mean much without it. Latency during a flash sale is not the same as latency at 4am on a Sunday morning.

Saturation is easy to miss, which is why I check it first. Queue depth and your connection pool to the payment gateway tend to fill up before latency moves. If your pool panel has been creeping upward for the last twenty minutes, then your p99 spike won’t come out of nowhere. Sometimes you can even alleviate the problem before checkouts are affected. In the screenshot above, the largest latency spikes line up with spikes in the pool.

Service health

The bottom section is for your engineers so they can troubleshoot root causes: CPU/memory per pod, count of pod restarts, GC pauses (if your runtime makes them available), and disk (if your service writes to disk). Root causes live down here, so treat this row with as much care as the ones above it, even if no one outside the team will ever scroll down that far.

Logs get three panels here. There’s log volume by level, because spikes in error logs will frequently show up before the error ratio moves, and a dramatic decrease in info logs is also usually something to investigate. The other two panels are live tails of the newest error lines and info lines. These save you from having to exec into a pod and cat them yourself. Typically, you’ll see the name of a failing payment gateway show up first in the error tail.

Alerts generated from this area should page very, very rarely. CPU may be at 85% with perfectly normal latency and errors. No users are affected, and nobody should be disturbed. This row is there to explain the rows above it, so let alerts from this row create tickets and leave the paging to the user section.

Reading top to bottom

The design places effect above cause intentionally. When triaging an incident, you scan down: the business section tells you how bad it is, the user section tells you what users are seeing, and the service health section tells you why it’s happening. The rest of the time, you scan back up. If someone sees high CPU and wants to double the fleet size, a flat latency graph and consistent checkouts will quickly settle that conversation.

Reading in either direction only works when each panel relates back to a question that someone will ask during an incident. I aim to keep each section between three and six panels, and move one-off debugging panels to their own dashboard. Then they can build up over there without getting in the way, and the core dashboard stays skimmable when something breaks.