Choosing a Distance Metric for Embeddings

The situation

A team has built a retrieval-augmented feature on Amazon Bedrock. Documents are chunked, run through an embedding model to produce vectors, and stored in an OpenSearch k-NNNearest-neighbour searchFinding the vectors closest to a query vector; at scale it’s approximated, trading a little accuracy for a lot of speed. index; at query time the user’s question is embedded the same way and the store returns the nearest chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. to stuff into the prompt. It worked in the prototype. In production, retrieval quality is oddly mediocre: the top hits are often loosely related rather than on the nose, and the relevant chunk that a human can find in seconds sometimes sits at rank forty instead of rank one.

Nothing is broken in the way monitoring understands broken. The index builds, queries return in single-digit milliseconds, there are no exceptions, and the embedding calls all succeed. When someone finally diffs the index settings against the model documentation, the culprit is one field: the index was created with the default space_type of l2, and the embedding model was trained and normalised for cosine similarity. The vectors and the search were speaking slightly different languages the whole time.

The underlying question is small and easy to get wrong: which distance metric should the vector store use, and how do you know it matches the model that produced the embeddings?

What actually matters

An embedding is a point in a few hundred or few thousand dimensions, and “similar” means “close” under some definition of distance. The definition is not free to choose after the fact. The embedding model learned its geometry during training against a specific notion of closeness, usually cosine similarity, and the numbers it emits only carry meaning under that same notion. The metric is not a tuning knob you turn for better results; it is a contract with the model, and the store’s job is to honour it.

The dividing line that decides the most is whether magnitude carries meaning. Cosine similarity looks only at the angle between two vectors and ignores how long they are, so a short vector and a long vector pointing the same way are treated as identical. Dot product (inner product) multiplies angle and magnitude together, so a longer vector scores higher for the same direction. Euclidean distance measures the straight-line gap between the two points, which blends direction and magnitude into one number and is dominated by magnitude when the lengths vary. For most text embedding models the magnitude is not where the meaning lives; the direction is. That is why cosine is the common default for text.

The point that turns this from trivia into a real failure is normalisation. A vector is normalised when it is scaled to unit length, so every vector sits on the same sphere and only its direction varies. Many embedding models return normalised vectors by design. Once every vector has length one, cosine similarity and dot product become the same computation, because the magnitude term is always one and drops out. Euclidean distance also lines up: on unit vectors, ranking by ascending Euclidean distance produces the exact same order as ranking by descending cosine similarity, because the two are tied together by a fixed relationship. So on normalised vectors, the three metrics agree on the ranking, and the choice barely matters.

The danger is the other case. If the model emits vectors that are not normalised, and the semantic signal lives in the direction, then Euclidean distance and raw dot product let magnitude differences distort the ranking. A chunk that happens to produce a longer vector can crowd out a more relevant chunk with a shorter one, or the reverse, purely on length that means nothing. This is the silent failure: the query runs, results come back, and they are subtly wrong in a way no error surfaces. RecallRecall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. drops and the only symptom is that the answers are worse than they should be.

The last thing that matters is where the choice actually lives. The metric is not set on the model; it is set on the vector store, at index-creation time, and it is sticky. In OpenSearch k-NN it is the space_type on the field mapping. In pgvector it is which operator you query with and which operator class the index was built for. Get it right when you create the index, because changing it later usually means rebuilding.

What we’ll filter on

1. Does the embedding model normalise its output, or return raw-magnitude vectors?
2. Does magnitude carry meaning for this model, or is the signal in the direction alone?
3. What metric was the model trained and documented for (the vendor’s stated recommendation)?
4. Which metrics does the target vector store expose, and what is its default?
5. Is the index setting a match for the model, or silently relying on a default that is not?
6. Is the same embedding path used for both indexing and querying, so the vectors are comparable?

The metric landscape

Cosine similarity. Measures the cosine of the angle between two vectors, ranging from -1 (opposite) through 0 (orthogonal) to 1 (identical direction). It ignores magnitude entirely, comparing only orientation. This is the default assumption for most text embedding models, including the Amazon Titan Text Embeddings and Cohere Embed families on Bedrock, because the training objective pushes semantically similar text to point the same way regardless of length. When in doubt on a text model, cosine is the safe first pick.

Dot product (inner product). Multiplies the vectors component-wise and sums, which folds both angle and magnitude into the score: same direction, longer vector, higher score. On raw vectors this is a different ranking from cosine. On normalised vectors it is identical to cosine, and it is slightly cheaper to compute because there is no length division, which is why several stores prefer inner product as the fast path for models that already return unit vectors. It is the right choice when the model documents inner product, or when the model normalises and you want the cheaper equivalent of cosine.

Euclidean (L2) distance. The straight-line distance between the two points, so smaller is closer. It is sensitive to magnitude, and unlike the two above it is a distance rather than a similarity, so the sort direction is reversed. L2 suits embeddings where absolute position and magnitude genuinely carry information, some image and spatial embeddings, rather than direction-only text vectors. Picking L2 for a cosine-trained text model is the classic mismatch that quietly wrecks recall on un-normalised vectors, and merely wastes the opportunity for the cheaper equivalent on normalised ones.

The stores expose the choice. In OpenSearch k-NN the metric is space_type on the vector field, with values including cosinesimil, innerproduct, and l2; the default is l2, which is exactly the trap in this scenario if you do not set it. In pgvector the choice is carried by the operator, <=> for cosine distance, <#> for negative inner product, <-> for L2 distance, and the index is built with a matching operator class (vector_cosine_ops, vector_ip_ops, vector_l2_ops) so the approximate index and the query agree. Bedrock Knowledge Bases sit on top of these stores, so the same rule applies to whichever vector store backs the knowledge base.

Side by side

The reading of the table for the scenario is direct: the model is a normalised, cosine-trained text embedder, so cosine (or its equal, inner product) is correct, and the L2 default that the index silently inherited is the mismatch. Because the vectors are normalised, switching to cosine will snap the ranking back into line.

The picks in depth

The scenario’s fix is to align the index with the model. The model returns normalised, cosine-trained vectors, so the store should search under cosine (cosinesimil in OpenSearch) or, equivalently, inner product, rather than the L2 default it silently inherited. Because the vectors are unit length, this is not a delicate rescue; cosine and dot product compute the same thing here, and even L2 would rank identically, so the moment the field is created with the right space_type the neighbours come back in sensible order. The lesson is that the default bit you, not that the maths is fragile. Never accept the store’s default metric without checking it against the model’s documentation.

For a model that returns raw, un-normalised vectors, the stakes are higher and there are two honest routes. Either query with the metric the model documents, cosine for a cosine-trained model, so magnitude is factored out where it carries no meaning; or normalise the vectors yourself before indexing and querying, at which point inner product becomes the cheap, correct choice. What you must not do is leave un-normalised cosine-vectors under an L2 index, because that is precisely the configuration where magnitude noise reorders the results and recall quietly collapses. If you normalise, normalise on both sides, indexing and querying, or the two are not comparable.

The reason this is worth care rather than a shrug is the failure signature. A wrong metric does not throw, does not slow the query, and does not show up in any health check; it just returns worse neighbours. The way you catch it is not monitoring but evaluation: a small labelled set of queries with known-relevant chunks, run through the pipeline, measuring whether the right chunks land in the top-kTop-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. . A metric mismatch shows up immediately as poor recall on that set, where it is invisible everywhere else. Build the eval set before you trust the retrieval, because it is the only instrument that sees this class of bug.

One more consistency trap sits underneath all of it: the same embedding model and the same normalisation must be used for both the indexed documents and the query. Embed the corpus with one model and the queries with another, or normalise one side and not the other, and the vectors live in incompatible spaces no matter how well the metric is chosen. The metric matches the model, and both sides of the search must run the same model.

A worked example: the L2-default index

The index was created without an explicit metric, so OpenSearch used its default. The mapping looked, in effect, like this:

PUT /chunks
{
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "l2"
        }
      }
    }
  }
}

The embedding model produces 1024-dimensional, normalised vectors trained for cosine similarity. Under l2 the search still runs and still returns ten neighbours, so nothing looks wrong, but the ranking is computed against a notion of closeness the vectors were never built for. Because these particular vectors are normalised, the damage is limited to using the wrong-but-equivalent metric; had they been un-magnitude-normalised, the same index would have returned genuinely misordered results.

The corrected mapping sets the metric to what the model documents:

PUT /chunks
{
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "cosinesimil"
        }
      }
    }
  }
}

The same choice in pgvector is made not on the column but on the query operator and the index that backs it: build the HNSWHNSWA graph-based vector index that walks neighbour links to find close vectors fast, at the cost of extra memory per vector. index with vector_cosine_ops and query with the <=> cosine-distance operator, so the approximate index and the search agree on the metric. In either store the vectors did not change and the model did not change; only the store’s definition of “near” was brought back into line with the model that created the vectors, and retrieval quality recovered without a single embedding being recomputed.

What’s worth remembering

1. The distance metric is a contract with the embedding model, not a tuning knob; the model learned its geometry against a specific notion of closeness, and the store must honour it.
2. Cosine measures angle and ignores magnitude, dot product blends angle and magnitude, and Euclidean measures straight-line distance and is dominated by magnitude when lengths vary.
3. Cosine is the safe default for text embeddings, including the Titan and Cohere families on Bedrock, because the signal lives in direction, not length.
4. On normalised (unit-length) vectors, cosine and dot product are the same computation and Euclidean ranks identically, so the choice barely matters.
5. The dangerous case is raw, cosine-trained vectors under an L2 index, where magnitude noise reorders results and recall drops with no error, slowdown, or alert.
6. The metric lives on the vector store, set at index-creation time: OpenSearch space_type (default l2), pgvector operator plus operator class; never accept the default without checking it against the model.
7. A wrong metric is a silent failure; the only instrument that catches it is a small labelled evaluation set measuring whether relevant chunks land in the top-k.
8. If a model returns un-normalised vectors, either query with the metric it documents or normalise the vectors yourself, and normalise both the indexed documents and the queries.
9. Use the same embedding model and the same normalisation on both sides of the search, or the vectors are not comparable no matter how well the metric is chosen.
10. Changing the metric usually means rebuilding the index, so choose it correctly when you create the field rather than discovering the mismatch in production.