Module 2: Embeddings and Vector Stores · RAG

How do you find the three relevant paragraphs in 10,000 documents in 50 milliseconds? Not by reading all of them and not by keyword matching. You convert every paragraph to a point in high-dimensional space, convert your query to the same space, and find the nearest points. That is what embeddings and vector stores do. By the end of this module you will understand how each step works and how to choose the right vector store for your system.

What Is an Embedding?

An embedding is a dense vector: a list of floating-point numbers that represents the meaning of a piece of text. The key property is that semantically similar text produces embeddings that are close together in the vector space, even if the words are completely different.

Consider these three sentences: "The patient has elevated blood pressure." / "The client's hypertension reading is high." / "The weather today is sunny and warm." An embedding model maps the first two sentences to points that are very close to each other in a 384-dimensional space, because they mean nearly the same thing despite sharing only one word ("is"). The third sentence maps to a point far from both.

This property comes from training. Sentence-BERT (Reimers and Gurevych, arXiv:1908.10084) introduced a training approach that fine-tunes BERT-style models using sentence pairs with known similarity scores. The model learns to pull semantically similar pairs toward each other and push dissimilar ones apart. After training, the embedding space organizes itself so that meaning, not word choice, determines distance.

Why dense vectors beat keyword matching for retrieval A keyword search for "hypertension" would miss a document that says "elevated blood pressure" even though they mean the same thing. An embedding search finds both, because their vectors are close in the semantic space. This is the core advantage of dense retrieval over BM25-style keyword scoring for knowledge-intensive Q&A.

Measuring Similarity

Once you have embeddings, you need a distance function. Three options appear most often in practice.

Cosine similarity measures the angle between two vectors. It ignores magnitude and focuses purely on direction. A cosine similarity of 1.0 means the vectors point in the same direction (identical meaning). A score of 0 means perpendicular (unrelated). A score of -1 means opposite directions. This is the most common choice for text similarity because the magnitude of an embedding reflects how many tokens were averaged, not how similar the text is.

Dot product combines angle and magnitude. For normalized vectors (magnitude = 1), dot product and cosine similarity give identical results. FAISS's IndexFlatIP uses inner product (dot product), so normalizing your embeddings before indexing makes it equivalent to cosine similarity.

Euclidean distance (L2) measures straight-line distance. It tends to work less well for text embeddings than cosine similarity, but it is the default for some vector stores and works acceptably when all vectors are normalized to the same magnitude.

Fig 2 · Text to Embedding to Vector Index

Approximate Nearest-Neighbor Search

An exact search over 1 million 384-dimensional vectors requires computing a dot product with every vector. That is 384 million multiplications per query, which takes seconds on a single CPU. For any real system you need approximate nearest-neighbor (ANN) search, which finds results that are very close to exact (typically 95-99% recall) in milliseconds.

Two indexing families dominate RAG systems today.

HNSW (Hierarchical Navigable Small Worlds) builds a graph where each node connects to its nearest neighbors. Search navigates the graph layer by layer, pruning unpromising paths. HNSW gives very high recall (usually above 99%) at low query latency, but builds slowly and uses substantial memory. The FAISS library (Johnson et al., arXiv:1702.08734) implements HNSW as IndexHNSWFlat. Weaviate and Pinecone also use HNSW as their default.

IVF (Inverted File Index) divides the vector space into clusters using k-means. At query time it searches only the clusters nearest to the query. IVF builds faster and uses less memory than HNSW, but recall drops if the query's nearest neighbors span cluster boundaries. IVF works well when combined with product quantization (IVF-PQ) to compress vectors further.

Choosing a Vector Store

The right vector store depends on your deployment constraints, not which one has the best marketing.

Store	Deployment	Best for	Tradeoff
FAISS	Local / in-process	Prototype and offline batch	No persistence, no metadata filtering built in
Chroma	Local or Docker	Small-team RAG, easy setup	Not designed for multi-node scale
Pinecone	Managed cloud	Production RAG, no infra team	Vendor lock-in, per-vector cost
Weaviate	Self-hosted or cloud	Hybrid search, rich metadata	More complex to operate
pgvector	Postgres extension	Teams already on Postgres	Slower ANN than dedicated stores at scale

Python · Embed with Sentence-Transformers, Index with FAISS

from sentence_transformers import SentenceTransformer
import faiss, numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')  # 384 dims

chunks = [
    "The patient has elevated blood pressure.",
    "Client's hypertension reading is high.",
    "Annual sales targets were exceeded this quarter.",
    "Revenue growth outperformed projections.",
    "The weather is sunny and warm today.",
    "Cloud cover expected this afternoon.",
]

# Embed all chunks
embeddings = model.encode(chunks, convert_to_numpy=True,
                           normalize_embeddings=True)

# Build FAISS inner-product index (= cosine sim after normalization)
dim = embeddings.shape[1]   # 384
index = faiss.IndexFlatIP(dim)
index.add(embeddings)

# Query
query = "Does the patient have high blood pressure?"
q_vec = model.encode([query], convert_to_numpy=True,
                      normalize_embeddings=True)

scores, ids = index.search(q_vec, k=2)
for score, idx in zip(scores[0], ids[0]):
    print(f"Score {score:.3f}: {chunks[idx]}")
# Score 0.831: The patient has elevated blood pressure.
# Score 0.748: Client's hypertension reading is high.

Try this

Open a Python notebook. Install sentence-transformers and faiss-cpu. Copy the code above and run it. Then add two sentences of your own from a domain you work in (legal, medical, financial, engineering). Run a query related to those sentences. Note whether the embedding model retrieves them correctly. This hands-on test will tell you more about embedding quality than any benchmark.