Managing Search Latency
At first I was only using BM25 to score the furniture catalog. I now have some business signals to reorder those results, and I use a hybrid query that searches my vector index as well and fuses the lists. Each one of these improvements has helped my search be a bit better, but also cost a little more.
You can get away with this if you have a small index. As your catalog grows, you’ll run into problems. Maybe a search last year would have returned 40k results. Now it’s returning 400k, and p95 is going to start increasing. No one deployment is to blame.
However, you can attribute most of this to one thing. You’re doing a calculation on every query that you could precalculate and store on your documents. Then you’d have a number stored on each document that the query can just read.
The cost
There are four function_score functions in the layered ranking query:
- One adds to your score if the product is in stock.
- Another adds to the score based on how much of that product has sold (a popularity curve on
amount_sold). - Another decreases your score based on how long the product has been around (freshness decay on
date_added). - If the user sets a target price, one more decays your score around it.
These all run on every document that matches your query, on every shard the query runs on, before there’s a top 10. So if 400,000 products match your query of record cabinet, that’s 1.6 million function evaluations to get you a top 10.
The more matches there are, the more expensive it gets. You won’t see a difference if you’re searching with tight filters. What will slow you down are broad queries with lots of matches, which is funny, because that’s what the majority of people search for.
It also costs you when using function_score, because it prevents block-max WAND from being used. Lucene is able to skip through most of those documents with this technique. It takes all the documents that match your term and splits them into blocks of 128. Then it records the highest score a document can get for that term in each block. While searching, Lucene knows the score of the current 10th result. If a block can’t score higher than the 10th result, it skips over all of those documents.
That’s why you’ll rarely see every hit scored by a basic BM25 query. The stored max scores only account for the text score, though. A document with a lower text score could be ranked higher because of popularity, so Lucene needs to score every doc in every block. The same goes for script_score, just even slower, since it has to run a script on every hit.
Denormalization
Price decay is the only rank function that will vary based on the query. For each user and page you will get the same number back from stock bonus, popularity curve and freshness decay if the query was record cabinet or walnut console. You are calculating a constant every time you make a query. Calculate those three values and save them to the document and you will have them available on every query.
It’s similar to a denormalized value in a database. You do extra work on write and take up a little more space, but you don’t have to do it on read. So you’ll want to add two fields to your mapping:
PUT /products_semantic/_mapping
{
"properties": {
"ranking_score": { "type": "rank_feature" },
"ranking_computed_at": { "type": "date" }
}
}
rank_feature was made for this. It holds any positive number that you want to factor into relevance. You could technically use a float with field_value_factor, but then you’re using a function, which scores all docs. With a rank_feature query, block-max WAND keeps working, because Lucene knows the max score that field can have for a given block, just like a term.
Two triggers
Both triggers run the same stored script:
PUT /_scripts/products-ranking-score-v1
{
"script": {
"lang": "painless",
"source": """
double score = 0.0;
if (ctx._source.stock_available != null &&
ctx._source.stock_available > 0) {
score += 2.0;
}
if (ctx._source.amount_sold != null) {
score += Math.log1p(ctx._source.amount_sold * 0.15);
}
if (ctx._source.date_added != null) {
long added = ZonedDateTime.parse(ctx._source.date_added)
.toInstant().toEpochMilli();
double ageDays = (params.now_ms - added) / 86400000.0;
double distance = Math.max(0.0, ageDays - 7.0);
score += 1.25 * Math.pow(0.5, Math.pow(distance / 30.0, 2));
}
ctx._source.ranking_score = Math.max(score, 0.001);
ctx._source.ranking_computed_at = params.now;
"""
}
}
It’s a copy of the function_score query in Layered Search Ranking. The stock bonus and popularity curve stay the same. I just hand-coded the freshness line from that query’s gauss decay, using the same numbers (origin: now, offset: 7d, scale: 30d, decay: 0.5).
The Math.max(score, 0.001) is there because rank_feature only accepts positive numbers. If the product has no stock and no sales, the first two values will be 0. Then the freshness value decreases over time, and once the product is about a year old, the value is too small for the field to hold and the update fails.
The first trigger is a schedule. Freshness keeps decaying whether or not you update your doc, so you want a job that recalculates old scores. This can be done with the same _update_by_query pattern we used in Painless Pipelines, just with a different script:
POST /products_semantic/_update_by_query?conflicts=proceed&wait_for_completion=false&slices=auto
{
"query": {
"bool": {
"should": [
{ "bool": { "must_not": { "exists": { "field": "ranking_score" } } } },
{ "range": { "ranking_computed_at": { "lt": "now-6h" } } }
],
"minimum_should_match": 1
}
},
"script": {
"id": "products-ranking-score-v1",
"params": {
"now": "2025-04-19T08:00:00Z",
"now_ms": 1745049600000
}
}
}
We pass in the time as an ISO string to set ranking_computed_at, and as epoch milliseconds for our math. Since freshness decays over the course of 30 days, a score that’s 6 hours old will be pretty close. You just need a job that runs frequently enough to keep up with freshness.
The second trigger is an event. If something changes a value that feeds into our score, like a stock adjustment or a batch of new sales, we can update that one document with the same stored script:
POST /products_semantic/_update/vinyl_record_cabinet_v3
{
"script": {
"id": "products-ranking-score-v1",
"params": {
"now": "2025-04-19T08:00:00Z",
"now_ms": 1745049600000
}
}
}
These both help us keep our field up to date:
flowchart LR
E["<strong>Feature change</strong><br/><span class='mermaid-detail'>stock, sales</span>"]
T["<strong>Timer</strong><br/><span class='mermaid-detail'>every 6 hours</span>"]
U["<strong>_update</strong><br/><span class='mermaid-detail'>one document</span>"]
UBQ["<strong>_update_by_query</strong><br/><span class='mermaid-detail'>stale documents</span>"]
F["<strong>ranking_score</strong><br/><span class='mermaid-detail'>rank_feature field</span>"]
S["<strong>Search</strong><br/><span class='mermaid-detail'>reads one number</span>"]
E --> U --> F
T --> UBQ --> F
F --> S
So we have the timer to take care of drift from the calendar, and the event to take care of changes to the data. Either one uses the same script, so your document receives the same score.
We can also grow our formula now. Maybe we want to add a margin curve next quarter, or replace the formula with the output of an offline model. We’d still be querying one value per document. That all happens in the update process and won’t impact our query time.
Query the score
When running your query, the functions array is combined into one rank_feature clause, as shown below in the lexical branch of the query from Hybrid Search:
POST /products_semantic/_search?search_pipeline=products-rrf
{
"size": 10,
"_source": {
"excludes": ["search_embedding"]
},
"query": {
"hybrid": {
"queries": [
{
"bool": {
"must": [
{
"multi_match": {
"query": "vinyl storage console",
"fields": ["title^4", "description"],
"type": "best_fields"
}
}
],
"should": [
{
"rank_feature": {
"field": "ranking_score",
"boost": 2.0
}
}
],
"filter": [
{
"range": {
"stock_available": {
"gt": 0
}
}
}
]
}
},
{
"knn": {
"search_embedding": {
"vector": [0.012, -0.034, 0.008, ...],
"k": 100,
"filter": {
"range": {
"stock_available": {
"gt": 0
}
}
}
}
}
}
]
}
}
}
Our query is much simpler now. We have a match and a filter, and one boost that’s capped. The boost is capped because of the saturation function, which rank_feature uses by default. It takes the value you stored and converts it to a boost between 0 and boost. The higher the value, the less extra effect it has. Previously, we used log1p and max_boost to accomplish this.
It’s a curve that bends around a typical stored score. A product with that score gets roughly half of the boost, and it diminishes as the score increases. Because the engine calculates that typical score from all the scores in the index, it stays accurate as your store grows.
Products that get filtered out by the in-stock filter still affect that typical score, though. So tons of old stock sitting at 0.001 can drag it down and push the boost near max for everything else. There’s also a chance a product’s boost will shift when other products’ scores change. If you find this happening with your judged queries, you can use a fixed pivot for the saturation function.
Rescore what’s left
The price decay is completely optional and only comes into play if the shopper specifies a target price, which you can’t compute ahead of time.
It doesn’t need to happen on every match either. You can use the rescore feature on your query, which only runs on the top results from your query. Add it when there’s a target price.
To keep it short, this is a lexical example. In a hybrid query, rescore would happen on each branch, and then RRF would combine the results:
POST /products_semantic/_search
{
"size": 10,
"query": {
"bool": {
"must": [
{
"multi_match": {
"query": "walnut record cabinet",
"fields": ["title^4", "description"],
"type": "best_fields"
}
}
],
"should": [
{
"rank_feature": {
"field": "ranking_score",
"boost": 2.0
}
}
],
"filter": [
{ "range": { "stock_available": { "gt": 0 } } }
]
}
},
"rescore": {
"window_size": 200,
"query": {
"rescore_query": {
"function_score": {
"functions": [
{
"gauss": {
"price": {
"origin": 1200,
"scale": 300,
"decay": 0.5
}
},
"weight": 0.75
}
]
}
}
}
}
}
window_size determines how many documents from each shard are rescored. So if you set your window size to 200, your gauss decay runs on 200 documents, not on every product that matches. You’ll want a window big enough to get your desired products on the page, but not so big that it’s cost prohibitive. 200 should work if you’re displaying 10 products per page.
If the product you’re trying to get ranked first isn’t in your main query’s top 200 results, your price decay won’t see it. You can always increase your window size, but the real fix is getting your main query to rank that product higher.
Smaller wins
Precomputing the score is the number one thing to do. Here are a few other tips:
- Put your yes-or-no conditions in
filter. Filters aren’t scored and they’re cached, so don’t put those inmust. track_total_hitsdefaults to 10,000. If you need the exact hit count you can set it totrue, but keep in mind it’ll need to visit every hit.- Use
search_afterfor paginating through results instead of a highfromvalue. If you’re going to page 100, each shard will rank 1,000 docs only to return you 10. - Make sure your
_sourcefields don’t include any that you don’t show on the page, such assearch_embedding. - Have fewer but larger shards. Each additional shard adds another result set to merge.
Evaluation
Once you have your new query up and running, test to see if it’s faster. Make sure it’s still ranking the same products well, too.
Start with latency. You can see how long a request took with the took value, which is in milliseconds. If you notice a request taking too long, look into the profile API. Get percentile measurements while running a decent amount of concurrency. Running one request on a warm cluster with no load will give you good numbers, but not your p99.
Then check relevance. Shifting math from query time to index time is still a ranking change because:
- The old query summed up the business signals and multiplied the text score by them.
rank_featuregives a boost with a ceiling. - The freshness part of a stored score can be up to 6 hours old.
Test your current query and the modified one using the judged-query loop. Your nDCG@10 should be about the same. If you see it decrease, retune your boost amount or hardcode your pivot. You should also keep an eye on your online metrics after making this change. You won’t be able to tell a bad page from a slow page with abandonment.
What we’d like is to do this work once when a document is written, not on every query. There will be thousands of queries for every update, so any constant we can pull out of the query saves us that work thousands of times over. As our catalog grows, our queries won’t, because we’re putting the ranking math into the index.