./ahmedhashim

In Offline Ranking Metrics, I talked about the typical ways of scoring a layered ranker against judged data. The resulting scores help you pick among candidate rankers before they’ve seen any production traffic. When one of these candidates actually goes live, however, you need some way to measure its effect.

Enter online evaluation. Online means the signal you care about is what users do after seeing the ranked page (links clicked, items purchased, etc.), which ultimately ties back to revenue and retention.

The event model

Each metric below is aggregated up from two IDs. A session_id represents a browser or signed-in session. A search_id represents one query you submitted and the resulting page. Even for metrics with “session” in their name, I take measurements at the search_id level first. Sessions will often have multiple searches you want to measure individually.

The backend must always log the search response. This provides a persistent record of what the ranker returned even if the user closes the browser before sending any interaction events. The frontend should log everything the user saw and did. Rendered impressions and clicks are an absolute minimum, but tracking dwell time and downstream conversions is valuable too. Impressions should correspond to result cards that were rendered on the page, not every hit returned by OpenSearch. The distinction between those two is important as soon as you try to correct for position bias.

Once you’ve got those events coming in, these metrics are pretty much just math.

Session abandonment rate

Session abandonment rate is the percentage of search sessions that don’t result in a click.

abandonment_rate = searches_with_no_click / searches_viewed

Imagine a customer searches our catalog for walnut record cabinet, looks at the results page, but doesn’t click on any product. That search would be considered abandoned.

This metric shows overall dissatisfaction, but doesn’t tell you what type of dissatisfaction. There could be stock-outs covering the top of the page. The query could be matching too much to be useful. Abandonment won’t tell you which of these situations is occurring, only that the page isn’t satisfying customers.

I like to monitor this metric two ways: once with all rendered searches included, and again subtracting out any searches which resulted in zero results. By subtracting out zero results, the second metric allows you to differentiate between ranking problems and recall problems. Let’s say I just tweaked negative_boost or popularity factor. Abandonment would be at the top of my list to check. If abandonment goes up but your zero-result rate stays the same, you can probably safely assume that the first page got worse.

Collapse the logs down to per-search records and the math is simple:

def abandonment_rate(searches: list[dict]) -> float:
    viewed = [s for s in searches if s["viewed"]]
    abandoned = [s for s in viewed if not s["clicked"]]
    return len(abandoned) / len(viewed) if viewed else 0.0

Click-through rate

Click-through rate is the share of impressions that get clicked:

ctr = result_clicks / result_impressions

CTR for search is most useful when broken down by rank + ranker version. Aggregate CTR hides position bias. Rank one is going to have more clicks than rank ten just because it’s first, even if rank ten is more relevant.

CTR is highly correlated with changes to how enticing results appear. Title/thumbnail change it. Showing price changes it. Pretty much any change you can make to a result’s appearance that doesn’t affect rank will change CTR. Correlated metrics are easily misconstrued, so that’s why I never look at it by itself, always beside something satisfaction correlated.

But really, for rank changes, I only care about the first page, and mostly the top five results. If you roll out something that helps rank eight at the expense of rank one, nobody’s going to notice.

As measured directly from impression/click data:

from collections import Counter

def ctr_by_rank(events: list[dict]) -> dict[int, float]:
    impressions, clicks = Counter(), Counter()
    for e in events:
        if e["type"] == "impression":
            impressions[e["rank"]] += 1
        elif e["type"] == "click":
            clicks[e["rank"]] += 1
    return {rank: clicks[rank] / n for rank, n in impressions.items()}

Session success rate

Session success rate measures the percentage of search sessions which were successful.

session_success_rate = successful_searches / searches_viewed

Success can be defined however you want, depending on your product. One reasonable heuristic to start with for this furniture catalog could be: we consider the search successful if the user clicked on a result and then viewed the product page for at least 30 seconds or added the product to their cart.

That heuristic won’t be perfect, but it will be a far better signal than clicks. You can trivially increase CTR by making your result cards more clickable without improving relevance in any way. If your success rate goes up, but CTR does not, that’s probably because the ranker is showing fewer, but more relevant options. Perfect!

Success rate tells you when the popularity weight on our layered ranker has drifted too far. If someone searching walnut record cabinet is bouncing off popular oak cabinets, then popularity was probably weighted too high or material was weighted too low.

Log why a search succeeded, not just if it did. It’s useful to have a success boolean for easy dashboarding, but it hard-codes whatever success rule you decide to use at the moment. If you keep track of dwell_ms (time spent on product page) and added_to_cart, you can adjust your definition of success later and re-derive success based on existing data, rather than backfilling.

Once successful is recorded on each per-search record, the rate appears identical to abandonment:

def session_success_rate(searches: list[dict]) -> float:
    viewed = [s for s in searches if s["viewed"]]
    successful = [s for s in viewed if s["successful"]]
    return len(successful) / len(viewed) if viewed else 0.0

Zero result rate

Zero result rate is the rate of searches which yielded zero results.

zero_result_rate = zero_result_searches / search_responses

This is the clearest of all recall metrics. When searching for walnut record cabinet brings back zero results for a shopper, there is no tweaking of a ranking algorithm that will salvage that search session. Either there are no products in the catalog that would match that shopper’s search, or the query isn’t being parsed adequately to retrieve them.

ZRR is likely the simplest metric to fix. Lots of high-volume zero-result queries are going to be missing synonyms or analyzer gaps that you can patch immediately. Say you index furniture catalogs. record console, vinyl storage, and LP cabinet all might mean the same thing.

Separate your unfiltered ZRRs from filtered ZRRs. Unfiltered ZRRs are almost always a recall/coverage problem that needs to be addressed. Filtered ZRRs are often perfectly reasonable, because it’s reasonable for some filters to return no matches.

Your most common zero-result queries will tell you more than the number itself ever will. If you have a ZRR of 2%, that’s fine if all of those misses are garbage queries. But if all of those misses are vinyl cabinet or walnut console, you have some indexing work to do.

ZRR is based on the raw logged search responses since result count is taken directly from the backend:

def zero_result_rate(responses: list[dict]) -> float:
    if not responses:
        return 0.0
    zero = sum(1 for r in responses if r["result_count"] == 0)
    return zero / len(responses)

Using them together

None of these numbers should ever be used to drive a ranking decision by itself. Each number highlights a different type of failure. If you are analyzing any one of them in isolation, chances are you are missing information from another.

Here’s the order I run through these tests when analyzing a ranking change:

1. ZRR always comes first. If recall is screwed up, it doesn’t matter what you do with the rest, because no amount of ranking is going to fix that page.
2. Abandonment is second, but only on queries that returned results.
3. CTR and success rate should be considered on every decision. Ideally both should be trending properly on page one.

Relationships > absolute values. When I see CTR increase but success rate drop, then I know the ranker is surfacing cards the user likes more, but worse matches. When ZRR moves and I haven’t changed query logic, I look in analyzers and filters first, because function_score is no magic voodoo that can fix a bad candidate set.

Online eval can only take you as far as your ability to relate metrics back to the shopper experience on the page. A shopper searches for walnut record cabinet and sees a ranked list of results. They click on a product and either convert or don’t find what they want. If you track that path cleanly, ranking changes are no longer guesswork.