Maddy Zhang explains retrieval-augmented generation, or RAG: answering questions with documents fetched at query time. Her eleven-minute overview follows the basic pipeline, its common failures and the upgrades used to address them. The emphasis is on choosing a retrieval technique for a specific problem, rather than treating a vector database as the finished application.

Supply current information at query time (00:46)

  • A model’s training data cannot contain a company’s private wiki or every recent update. Zhang contrasts retraining a model with fetching relevant information when the user asks a question.
  • RAG puts those retrieved passages into the model’s context. The application can update its source documents without incorporating each change into the model’s weights.
  • This makes the source material available for citation. The generation step still depends on which passages the system actually retrieves and supplies.

Follow the indexing, retrieval and generation stages (02:49)

  • Indexing: split documents into chunks, then convert them into embeddings, numerical representations of meaning. Store those representations in a searchable database.
  • Retrieval: embed the question using the same embedding model. Search for nearby representations to find passages with related meanings, even when they use different words.
  • Generation: place the retrieved chunks beside the original question in the prompt. Ask the language model to answer from that material and identify the passages it used.

Diagnose what the basic pipeline misses (04:39)

  • Chunk boundaries matter. Small chunks can lose surrounding context. Large ones can mix several topics and weaken the search signal. Zhang recommends respecting sentences, paragraphs and section headings when splitting documents.
  • Exact identifiers need literal matching. Error codes, product numbers and internal acronyms can be poor semantic-search targets. Her example is an engineer searching for error 4092 in a checkout service and receiving thematically related but incorrect passages.
  • More context can obscure the answer. Zhang describes the “lost in the middle” problem: relevant information buried among many chunks may receive less attention than material at the beginning or end.

Combine literal search with meaning, then rerank (06:32)

  • Hybrid retrieval runs vector search alongside BM25, a keyword-based ranking method. One searches for related meaning; the other preserves exact terms.
  • Reciprocal rank fusion combines their ordered result lists without requiring their different numerical scores to share a scale.
  • A reranker makes a second pass over the candidates. Zhang describes a cross-encoder, a model that considers the question and a candidate passage together, to judge relevance more carefully.
  • Her example retrieves 50 candidates, reranks them and sends the best five to the answering model. This keeps the final context smaller and more focused. These counts illustrate a pipeline, rather than a universal setting.

Rewrite questions when they are poor search queries (08:16)

  • Conversational questions contain pronouns, implied context and ambiguous references. A model can rewrite them into explicit search requests or split them into focused subquestions.
  • HyDE, hypothetical document embeddings, generates a possible answer first and uses it to guide retrieval. The idea is that answer-like text can resemble the desired documents more closely than the question does.
  • Zhang notes the extra model call and delay. Query transformation is a trade-off to tune for the application.

Repeat retrieval for questions spanning several documents (09:07)

  • Zhang’s example combines refund policy, promotional terms and an enterprise account’s signup date. A single search may miss part of the evidence.
  • Agentic RAG puts retrieval inside a loop: plan searches, retrieve, evaluate whether the results are sufficient, then search again if needed.
  • She closes by stressing failure analysis and evaluations, repeatable tests of system behavior. Understanding which upgrade addresses which failure matters more than reciting the basic pipeline.

Caveat: this is a conceptual overview, not a tested implementation or a guarantee that these upgrades improve every corpus; HubSpot sponsored the video.