Semantic Search
Traditional ranking used in Layered Search Ranking involves matching words in the query with words in the document. The query walnut record cabinet should match documents containing those words in their title or description fields.
The challenge comes when users search for things using different words than what you might expect. A shopper typing vinyl storage console may still be interested in the product Vinyl Record Cabinet. A support user searching for reset my login may need the article titled Changing your password. Synonyms and carefully crafted analyzers can help you connect these dots, but those rules quickly turn into another system to upkeep.
Semantic search matches on meaning using embeddings. A model trained over vast swaths of text produces a vector for each document/query. Text that occurs in similar contexts is embedded close together. Search can then return nearest neighbors of the query so reset my login will match Changing your password with zero word overlap.
Embeddings
An embedding is essentially a vector of floats (a fixed-length list of numbers) created by a model. Texts that have similar meanings should be close to each other.
For a product catalog, this text:
Title: Vinyl Record Cabinet
Materials: walnut, iron
Description: Mid-century inspired storage cabinet for vinyl records with sliding doors and adjustable shelves.
should land close to queries like:
record storage
vinyl cabinet
walnut media console for LPs
It’s just an array of floats stored in the index. Here it’s shortened to save space:
embedding = [
0.0124,
-0.0341,
0.0087,
0.0219,
-0.0146,
0.0032,
# ...many more dimensions
]
In order to produce that vector, the model first tokenizes the text. Each family of models has its own tokenizer. Tokens typically represent whole words or parts of words:
text = "Vinyl Record Cabinet"
# Illustrative only. Each model family has its own tokenizer.
tokens = ["Vinyl", " Record", " Cabinet"]
token_ids = tokenizer.encode(text)
Each model has a certain context length. This is the maximum amount of text that can be put into one input. Context length is measured in tokens, not characters or words. This means that a sentence that fits into one model may be too long for another.
Size of output is not dependent on size of input: a 1024-dimensional model will output 1024 numbers for a one-word title and 1024 numbers for a full paragraph description.
The model has no knowledge of your catalog. It simply converts text to numbers. Storage of those vectors, and retrieval of nearest neighbors that pass your filters, is handled by the search engine.
Similarity will typically refer to one of cosine similarity, dot product, or Euclidean distance. Choose whichever metric your model was trained with (the docs should state which one). In OpenSearch this is referred to as the space_type on your knn_vector field: cosinesimil, innerproduct, or l2 for Euclidean distance. The example index mapping below uses cosinesimil. Note that the mapping locks you into both the dimension and distance function. You cannot swap models out later without re-embedding and reindexing all of your documents.
Choosing a model
Start with your data. Yeah, leaderboards such as MTEB are great for helping you weed out models, but your model will be scoring against your corpus of data. A high performing model on sentence similarity straight out of the box probably doesn’t know your internal company abbreviations or product names translated into other languages.
A general-purpose text model is a fine baseline for catalogs and support docs. But if your corpus is made of code or legalese, test a domain-specific model with your same queries. The bigger generic model isn’t always better.
Then there’s the matter of where the model is running. It’s easiest to start with a hosted API, since you don’t have to host model servers yourself. Self-hosting your model will cost less per embedding and keep your data on premises.
One decision you should make early in your planning is whether you want your search functionality to be symmetric or asymmetric:
- Symmetric: queries and documents closely resemble each other (e.g. finding duplicate questions).
- Asymmetric: queries are shorter than the documents they retrieve (e.g. a shopper query vs. a product description, or a user question vs. a KB article).
See Sentence Transformers for a nice explanation of the difference.
When doing asymmetric search (you have queries and documents and want short queries to be closer to documents that answer them), label which input is a query and which input is a document if your model allows. Some hosted APIs take this as a parameter (Cohere for example). Open models like E5 just assume you put a query: or passage: prefix on the text to embed. If you fail to provide these labels and your model requires them, your results will silently degrade.
Models also differ by how many dimensions their output vectors have. Small open models like MiniLM output vectors with 384 dimensions, whereas OpenAI’s large embedding model produces embeddings with 3072 dimensions. Many models are in between those two extremes, but 768 and 1024 are common.
The number of dimensions you choose will dictate what your stored index looks like. Lower-dimensional indices like 384 are cheaper and will perform faster queries, but they may not store enough information about longer or more domain-specific texts. Higher-dimensional indices like 3072 provide more “space” for your retriever to differentiate between different texts, but cost more memory per query.
Certain models have been trained with Matryoshka representation learning, allowing leading dimensions of a truncated vector to work correctly out-of-the-box. OpenAI exposes this functionality through a dimensions parameter. As long as your truncated vector scores comparably to the full vector on queries you care about, you can use a shortened vector without needing to switch models.
Storage tradeoffs
Vectors actually require more space than one might think:
1,000,000 documents * 1,536 dimensions * 4 bytes = 6.1 GB
1,000,000 documents * 3,072 dimensions * 4 bytes = 12.3 GB
That’s FP32: 32 bits / 4 bytes per dimension. We haven’t even counted the HNSW graph itself, replicas of it, or the copy stored in _source.
Lower precision values take up fewer bytes per dimension:
FP32 32 bits 4 bytes per dimension
FP16 16 bits 2 bytes per dimension
bfloat16 16 bits 2 bytes per dimension
FP8 8 bits 1 byte per dimension
int8 8 bits 1 byte per dimension
binary 1 bit 1/8 byte per dimension
For the exact same 1M documents with 1,536 dimensions, that ends up being:
1,536 dimensions * FP32 = 6.1 GB
1,536 dimensions * FP16 = 3.1 GB
1,536 dimensions * int8 = 1.5 GB
1,536 dimensions * binary = 192 MB
FP16 and bfloat16 both cut vector storage in half, but they make different tradeoffs:
FP16stores more digits per number. This lets it distinguish between0.01240and0.01242. The downside is that it can’t represent numbers larger than roughly 65,000.bfloat16stores fewer digits per number. The upside is that it can represent numbers up to the same size asFP32can.
Embedding values are small, and when ranking you’re differentiating by small amounts between these values. So, FP16 is more suited to stored vectors. You’ll see bfloat16 more inside models, where values can get larger than FP16 can represent.
What formats you have available will depend on your index. OpenSearch 2.19 supports storing FP16 vectors using Faiss, one of its vector engines. It also supports int8 as well as binary vectors. It doesn’t support storing bfloat16 or FP8. Some APIs, like Cohere’s, may return int8 or binary embeddings directly.
Smaller vectors are less accurate. Lower precision flattens out the distance amongst nearest neighbors, and fewer dimensions mean less signal overall. However, if two configurations get roughly the same score on nDCG@10 for your judged queries, err on the side of the cheaper one. Retrieve more candidates than you need with the small format, then re-score them using the full-precision vectors before trusting the top candidates.
Regardless of your choice, save the model ID, tokenizer, dimension, precision, distance metric and input template with every embedding you create. If you ever change one of them, you’ll have to create a new vector field/index.
Indexing embeddings
There’s one big decision about indexing, though: what text to embed. In the case of a product, I create a small chunk of text based on the fields that describe it:
def product_embedding_text(product: dict) -> str:
return "\n".join(
[
f"Title: {product['title']}",
f"Materials: {product.get('materials', '')}",
f"Description: {product.get('description', '')}",
]
)
For the record cabinet, that creates one string that then becomes one vector:
Title: Vinyl Record Cabinet
Materials: walnut, iron
Description: Mid-century inspired storage cabinet for vinyl records with sliding doors and adjustable shelves.
Do not embed values such as price, inventory count, or sales count into the embedding text. Leave those as filterable/sortable structured fields such as stock_available and amount_sold, as seen in the mapping below. Place the vector field immediately next to those fields:
PUT /products_semantic_v1
{
"settings": {
"index": {
"knn": true
}
},
"mappings": {
"properties": {
"title": { "type": "text" },
"description": { "type": "text" },
"materials": { "type": "keyword" },
"stock_available": { "type": "integer" },
"amount_sold": { "type": "integer" },
"embedding_model": { "type": "keyword" },
"embedding_text_hash": { "type": "keyword" },
"search_embedding": {
"type": "knn_vector",
"dimension": 1536,
"method": {
"name": "hnsw",
"engine": "lucene",
"space_type": "cosinesimil",
"parameters": {
"m": 16,
"ef_construction": 100
}
}
}
}
}
}
The vector-specific portions of the mapping definition:
"knn": trueenables vector search on the index.knn_vectoris the field type that stores the embeddings. Itsdimensionmust match the model, 1536 fortext-embedding-3-small.hnswis the typical approximate nearest neighbor algorithm.mcontrols how many links each vector is connected to in the graph andef_constructioncontrols how thoroughly the graph is constructed. Both values are the OpenSearch defaults.luceneshould be used as the engine if your queries are filtered, because the filter is applied during the graph search.
embedding_model and embedding_text_hash tell you which model was used to generate each vector and what text it came from, allowing you to find stale vectors after upgrading models.
Point a stable alias at the index so you never hard-code an index name in queries:
POST /_aliases
{
"actions": [
{ "add": { "index": "products_semantic_v1", "alias": "products_semantic" } }
]
}
Next, batch-generate embeddings and index them alongside each product:
from hashlib import sha256
from openai import OpenAI
from opensearchpy import OpenSearch, helpers
client = OpenAI()
search = OpenSearch(hosts=[{"host": "localhost", "port": 9200}])
EMBEDDING_MODEL = "text-embedding-3-small"
def embed_batch(texts: list[str]) -> list[list[float]]:
response = client.embeddings.create(
model=EMBEDDING_MODEL,
input=texts,
)
return [item.embedding for item in response.data]
def bulk_actions(products: list[dict]):
texts = [product_embedding_text(product) for product in products]
embeddings = embed_batch(texts)
for product, text, embedding in zip(products, texts, embeddings):
yield {
"_op_type": "index",
"_index": "products_semantic_v1",
"_id": product["id"],
"_source": {
**product,
"embedding_model": EMBEDDING_MODEL,
"embedding_text_hash": sha256(text.encode()).hexdigest(),
"search_embedding": embedding,
},
}
for i in range(0, len(products), 100):
helpers.bulk(search, bulk_actions(products[i : i + 100]))
Execute the same code any time products are created or updated. If you modify the model or template, handle it just like any other reindex: launch products_semantic_v2, backfill embeddings, compare its quality to v1, then swap the products_semantic alias over to the new index.
There are two errors you want to detect early:
- Vectors from different models in the same vector field. You cannot compare vectors made with different models. If half your documents were indexed with one model and half with another, you’ll end up with nonsensical distances between them.
- Documents that are too long. Some APIs will reject documents that exceed the context length, while others will truncate the end without telling you. That’s why the template above puts the title and materials section before the description section. Anything truncated will be from the least important section. You’ll need to chunk really long documents like support articles into separate paragraphs with their own vectors. This is a much more significant change to your index.
Vector-only retrieval
At query time, generate the user’s query embedding with the same model, then use that vector to search for the nearest documents:
GET /products_semantic/_search
{
"size": 10,
"_source": {
"excludes": ["search_embedding"]
},
"query": {
"knn": {
"search_embedding": {
"vector": [0.012, -0.034, 0.008, ...],
"k": 100,
"filter": {
"range": {
"stock_available": {
"gt": 0
}
}
}
}
}
}
}
The query vector above has been truncated for clarity. It would have the same dimension as the stored vector field in production. The excludes property prevents OpenSearch from returning each stored vector for each document in the results.
For the index above, you have two knobs to adjust:
kcontrols how many nearest neighbors vector search will find initially. OpenSearch then filters these down tosizeresults.ef_searchcontrols how many candidates HNSW maintains while traversing the graph. The longer this list is, the more vectors will be compared before the topkare chosen. Lucene picks whichever is larger of these two values, so settingef_searchlower thankis ineffective.
Other setups add their own settings. An IVF index partitions similar vectors into buckets when created, and nprobes determines how many of the nearest buckets are searched for a query. For quantized vectors, oversample_factor determines how many candidate nearest neighbors to re-score at full precision.
Increasing any of these values improves recall at the expense of additional CPU and latency. These parameters should be tuned against a judged query set, as described in the evaluation section below.
Vector-only retrieval excels with similar-item pages and natural-language search over long-form documents. It can also be effective when the user vocabulary is unpredictable.
Where vector search struggles is exact matching. A model might think that oak media console is similar to walnut record cabinet, even though the wood doesn’t match. SKUs/model numbers suffer too, since they’re not natural language.
Threshold and filtered retrieval
Top k retrieval will always return k neighbors if there are enough documents to return. But sometimes you’re returning “similar products” or “related articles”, and a weak neighbor might be worse than just returning fewer items. OpenSearch supports radial search to solve this: instead of requesting the top k, you specify a minimum score with min_score or maximum distance with max_distance and retrieve all documents that pass it.
Thresholds work well for recommendations and deduplication. They don’t work as well for primary search since users typically expect to see an entire page of results.
Thresholds are hard to tune though. The threshold that works for one query will often be too strict or too lenient for another query. This is especially true if your queries vary widely in length. The threshold also only applies to one model + text template. A vector returning a cosine score of 0.82 from one model is not directly comparable to 0.82 from a different model, meaning you have to re-tune it if you change either of those.
Ok, so if a user filters on walnut or “in stock”, that should be an exact filter like it would be with a lexical query. You want a vector query for the semantics, but exact filters for hard facts.
Notice where the filter is. The last query had it nested down under knn. It’s been pulled out to post_filter:
GET /products_semantic/_search
{
"size": 10,
"query": {
"knn": {
"search_embedding": {
"vector": [0.012, -0.034, 0.008, ...],
"k": 10
}
}
},
"post_filter": {
"range": { "stock_available": { "gt": 0 } }
}
}
Order matters. With this query OpenSearch first finds the 10 nearest products, then filters out-of-stock products from those results. If 7 of the top 10 products are out of stock, 3 results remain. Product 11 and Product 12 may have been in stock, but your search cut off at 10.
If you place the filter within knn, OpenSearch will skip out-of-stock products during search. It will keep searching until it finds 10 products that are in stock.
Filtering within knn can also be faster if the filter is selective. If it cuts down the index to a small subset of the original size, it’s faster to score the query against each document than it is to traverse an HNSW graph full of results you don’t care about. The Lucene and Faiss engines automatically pick whichever strategy is faster on a per-query basis.
Evaluation
It’s easy to feel like an embedding index is doing a great job if your few demo queries perform well, but demo queries are often hand-picked queries where embeddings excel. Run your judged query set against each embedding configuration instead. Break down nDCG@10 and MRR by query type as you would with offline ranking metrics. Also evaluate recall at k, because nDCG and MRR won’t tell you if the correct products were included in the candidate set to begin with.
Review your misses prior to changing the alias. If walnut record cabinet is returning oak consoles too often, there might be a problem with your text template, model selection, or precision setting. If SKUs/model numbers matter on that page, decide whether they should be embedded in the text or kept for exact matches outside the vector query.
Once your vector index goes into production keep an eye on your online ranking metrics. Zero-result rate will dive to almost zero, but that’s not a useful signal, because vector search will always return its top k nearest neighbors. Abandonment rate should go down on queries that used to return nothing useful due to vocabulary mismatch. Click-through rate may go up for undesirable reasons, so look at it with success rate before concluding the update had a positive effect.
Most of your persistent work will actually go into the index itself. Thoughtfully select your model, dimension and precision, and index enough metadata to rebuild the field. After that point, modifications to any of those variables are just another reindex you can monitor.