./ahmedhashim

Each OpenSearch index is composed of shards. Once set at index creation time, the number of shards cannot be adjusted. So far in this series, we’ve built our furniture catalog search engine with hybrid ranking and a fast query. But its index products_semantic_v1 only has the single shard OpenSearch allocated to it by default. Let’s choose an appropriate number of shards.

A tiny catalog requires no more than a single shard, but by the time you need more than one shard, the index is large and slow to split. Choosing the number is mostly a capacity planning exercise, but it also affects the scores that are returned.

One shard by default

Each shard is an independent search engine (a complete Lucene index with its own segments and caches) that is only aware of the documents in its partition of the catalog. Given a query, each shard executes it independently and produces a scored list.

OpenSearch creates indexes with index.number_of_shards: 1 by default. That works well for a small index because each query is processed in exactly one place with nothing to merge.

Fan-out

If there is more than one shard, searching works like this:

1. The coordinating node queries one copy of each shard.
2. Each shard reads through its segment and returns its local top size hits.
3. The coordinator combines those lists into a global ranking.
4. It retrieves the full documents for the final result set.

  flowchart TD
    Q["<strong>Query</strong><br/><span class='mermaid-detail'>walnut record cabinet</span>"]
    C["<strong>Coordinator</strong>"]
    S0["<strong>Shard 0</strong><br/><span class='mermaid-detail'>top 10</span>"]
    S1["<strong>Shard 1</strong><br/><span class='mermaid-detail'>top 10</span>"]
    S2["<strong>Shard 2</strong><br/><span class='mermaid-detail'>top 10</span>"]
    M["<strong>Merge</strong><br/><span class='mermaid-detail'>30 candidates, top 10 out</span>"]

    Q --> C
    C --> S0
    C --> S1
    C --> S2
    S0 --> M
    S1 --> M
    S2 --> M

Each additional shard adds parallelism at the cost of overhead. For example, according to the Elasticsearch sizing guidance, searching 1,000 shards of 50 MB each is substantially more expensive than searching a single shard of 50 GB holding the same data.

Custom _routing lets a query hit only a single shard if you know each query only needs data for a single customer. But a shared catalog of furniture doesn’t work like that, so each search request hits every shard. More shards means more answers to wait for on each search.

Too many shards

Each shard incurs memory and cluster bookkeeping overhead even when it contains very little data. Each query must also be executed once per shard, so adding shards causes every search to do more work.

Beyond some point, additional shards only add overhead and don’t make queries any faster. Ten 6 GB shards store the same data as three 20 GB shards, but the coordinator now has to merge ten lists instead of three, and the cluster has to keep track of ten sets of segments.

Too few shards

If a node dies, the cluster must completely rebuild each of its shards from scratch on another node. A 60 GB shard takes quite a while to copy around, and leaves the cluster with reduced redundancy while it’s in progress. And because each shard is searched on a single node, if you have a one-shard index OpenSearch runs every query on a single node, even if you have six available.

Elastic’s usual guidance recommends shards be 10 to 50 GB each: large enough that you don’t have too many shards, but small enough that if a shard is lost, you can rebuild it quickly.

The sizing math

We’re up to five million products in our catalog now. If we apply the embedding storage math, here’s how that space breaks down:

5,000,000 products

text and metadata, _source  ~5 KB each        25 GB
vectors, 1,536 dims, FP32   6.1 KB each       31 GB
vectors as JSON, _source    ~12 KB each       60 GB
HNSW graph, m = 16          128 bytes each    0.6 GB
                                              ------
                                              ~117 GB

Each vector is stored twice. knn_vector contains the binary copy that search operates on, and _source contains the original JSON array. That’s ~24 KB of decimal text that compresses to about half the size. If we dropped it from _source we’d save 60 GB, but reindexing reads _source, so the next shard change would have us embed the entire catalog again.

Vectors account for about three quarters of the index size. This is unsurprising for a catalog that stores embeddings. The point is our shard size is dictated by data shoppers will never see.

One shard at 117 GB is already more than double the 50 GB limit. Three shards would put us at 39 GB each, which is within the limit but close enough to the upper range that a year’s growth would surpass it. Six shards puts us around 19.5 GB each with two primaries per node on three search nodes. That should give us enough headroom for the catalog to grow.

PUT /products_semantic_v2
{
  "settings": {
    "index": {
      "number_of_shards": 6,
      "number_of_replicas": 1,
      "knn": true
    }
  }
}

Mappings are exactly the same as v1. Only the settings changed.

With reindexing complete, this is what _cat/shards looks like now:

GET /_cat/shards/products_semantic?v&h=index,shard,prirep,store,node

index                 shard  prirep  store    node
products_semantic_v2  0      p       19.6gb   search-node-1
products_semantic_v2  0      r       19.6gb   search-node-2
products_semantic_v2  1      p       19.3gb   search-node-2
products_semantic_v2  1      r       19.3gb   search-node-3
products_semantic_v2  2      p       19.8gb   search-node-3
products_semantic_v2  2      r       19.8gb   search-node-1
products_semantic_v2  3      p       19.4gb   search-node-1
products_semantic_v2  3      r       19.4gb   search-node-3
products_semantic_v2  4      p       19.5gb   search-node-2
products_semantic_v2  4      r       19.5gb   search-node-1
products_semantic_v2  5      p       19.2gb   search-node-3
products_semantic_v2  5      r       19.2gb   search-node-2

Each node has two primaries, and each replica resides on a different node from its primary. A broad query now runs on three nodes simultaneously instead of one.

Resizing

It’s not possible to change the shard count of an index after creation. OpenSearch provides the _split and _shrink APIs, which clone an index into a new index with more or fewer shards, respectively. However, both require you to disable writes to the original index first.

If you’re working with a catalog that can’t have writes disabled, you can use live reindexing: create the new index with the appropriate number of shards, reindex into it, then point the alias at the new index. You don’t need to change anything else, as all queries already go through the alias.

Shards change scores

BM25 stats are kept per shard. The default query_then_fetch search type then scores documents using each shard’s local term and doc frequencies. So the IDF for a term on shard 0 is really not the IDF for that term on shard 2. But hashing 5 million docs into shards by the default _id gives statistically similar shards, so no one bats an eye. It shows up when you have small or skewed indexes. A classic example is a staging cluster with 2% of your production data, where the same query (literally the same code) comes back in an obviously different order.

dfs_query_then_fetch fixes this by collecting global frequencies in an extra round trip first:

GET /products_semantic/_search?search_type=dfs_query_then_fetch

That extra roundtrip means slower queries, so I use it only for debugging. If search hits come back in different orders between these two search types, your shards are too small or your routing is skewed.

The vector branch has the same problem but manifests differently. k from the k-NN query is applied per shard. Each shard calculates the k nearest neighbors across its own graphs, then returns its top size choices up to the coordinator node to merge and trim to the top size. When a hybrid query runs, it uses k: 100. One shard? That’s 100 candidates. Your query matches on six shards? Now you’re doing k: 100 six times, but the coordinator node only keeps ten out of those sixty results. More candidates means slower queries, but can mean better recall.

Note that adjusting the number of shards can produce different results, even without modifying your query. Keyword scores change a little, vector results change more. RRF blends what the two produce into your final page.

Replicas

Sometimes you’ll hear people say that you can increase performance just by adding replicas. This is rarely the case. Since the query only reads one copy of each shard, replicas will not make an individual query execute any faster. They do let the cluster handle more queries in parallel, though. They also ensure that a copy will be available if a node fails.

The replica count can be adjusted at any time, unlike the shard count:

PUT /products_semantic/_settings
{
  "index": {
    "number_of_replicas": 2
  }
}

Storage grows accordingly: 117 GB of primaries grows to 351 GB on disk with two replicas.

Evaluation

Changing the shard count requires two validations, one for speed and one for ranking. The ranking validation can be easily overlooked, because the change seems to only be ops work.

Benchmark latency with representative production concurrency before and after your change, including the p99. Adding shards may decrease the p50 of wide queries, but the extra overhead of merge workloads can make your slowest queries have worse tail latency.

Then repeat the judged-query loop. You modified BM25 scores and vector candidates, but total nDCG@10 shouldn’t move much with five million products. If it does move significantly, troubleshoot the vector branch first. After the alias swaps, watch online metrics like you would with any ranking deploy.

One shard is OK early on, and resizing is easiest when your index is small. Choose whatever number is logical for you, but write down why you chose what you did so you’ll remember when your catalog grows past it. Then, when you do need to resize it, keep an eye on ranking as well as latency.