LLM application engineering

What RAG adds to a model call

Suppose a support assistant must answer from a private policy manual. A plain generation request gives the model a question and instructions. A retrieval-augmented generation request first searches the manual, adds relevant passages to the request, and asks the model to answer from that context.

The application still owns the pipeline. The search system returns passages and scores. The language model composes the response.

user question
  -> retrieve relevant chunks
  -> put selected chunks in the model context
  -> generate an answer tied to those chunks

Wrong "RAG retrains the model on my documents."

Right Retrieval supplies context for the current request. OpenAI's current Retrieval API documentation describes vector stores as searchable indices whose files are chunked, embedded, and indexed. It does not describe retrieval as a model-weight update.

Embeddings and vector search

An embedding is a vector representation of an input. Semantic search compares the query vector with stored vectors to find conceptually similar text. OpenAI's retrieval documentation shows why this differs from keyword matching: a relevant result can share few or no words with the query.

A vector database or vector-capable search index stores vectors with the source text and useful metadata. A typical record needs enough information to return evidence, not only an array of numbers:

{
  "chunkId": "refunds-7",
  "text": "Refunds are available within 30 days with a receipt.",
  "embedding": [0.012, -0.031, 0.008],
  "source": "returns-policy.md",
  "region": "EU"
}

The three embedding numbers are illustrative. Real vector length depends on the embedding model.

Wrong "The nearest vector is guaranteed to contain the answer."

Right Vector search returns candidates ranked by similarity. OpenAI's search response includes chunks, similarity scores, and the source file. The application still decides which results are relevant enough to send to the model.

Why documents are split into chunks

Search often works better when a result points to a focused passage. Azure AI Search documents several approaches, including fixed-size chunks with overlap, boundaries based on sentences or headings, and semantic chunks that preserve relationships across sentences and paragraphs.

Chunking also keeps embedding inputs and generated prompts within model input limits. A whole handbook may fit a model limit yet still be poorly represented by one vector because it covers unrelated subjects.

Overlap can preserve text that crosses a boundary. More overlap also duplicates content in the index and in retrieved results, so it is a parameter to test rather than a universal percentage.

Grounded answers can still fail

Grounding uses retrieved passages as the source material for a response. Microsoft Foundry's groundedness evaluator checks whether a generated response aligns with the supplied context without fabricating content.

The pipeline can still fail before or during generation. It may retrieve the wrong passage, omit the right one, or produce a claim that the supplied passage does not support. Those are different failures. The middle notes show how hybrid search, reranking, filtering, and context selection address them.

Chunk boundaries change what can be retrieved

A fixed token count is easy to implement, but it can separate a heading from its paragraph or split a procedure between two chunks. Structure-aware chunking uses sentence, line, Markdown, or HTML boundaries. Azure AI Search also documents semantic chunking that keeps related sentences and paragraphs together.

Preserve the source identity and headings alongside each chunk. The model needs the passage for answering, while the application needs metadata for filtering and source references.

type IndexedChunk = {
  id: string;
  text: string;
  documentId: string;
  headingPath: string[];
  updatedAt: string;
  tenantId: string;
  embedding: number[];
};

Choose chunk parameters with a query set from the real workload. Azure's RAG guidance recommends experiments because the useful configuration depends on the data and expected queries.

As of 2026-07-12, OpenAI vector stores default to chunks of 800 tokens with 400 tokens of overlap. The documentation allows a chunk size from 100 through 4096 tokens and says overlap should not exceed half the chunk size. These are current service settings, not general chunking recommendations.

for each candidate chunk size and overlap:
  index the same corpus
  run the same labeled queries
  compare retrieval coverage, rank quality, latency, and context use

Wrong "More overlap always improves retrieval."

Right Overlap can keep boundary context, but it repeats material. Compare configurations on fixed queries instead of assuming that the largest allowed overlap wins.

Semantic, keyword, and hybrid retrieval

Vector search handles conceptual similarity and paraphrases. Exact strings such as product codes, dates, names, and specialized terms can favor keyword search. Azure AI Search therefore runs keyword and vector queries in parallel for hybrid search, then merges their ranked lists with Reciprocal Rank Fusion.

This matters when one corpus supports both question styles:

"Why was my login rejected?"  -> vector retrieval can match a policy explanation
"ERR_AUTH_041"                -> keyword retrieval can preserve the exact code

Wrong "Embeddings make keyword search obsolete."

Right The two methods recover different candidates. Hybrid retrieval keeps both signals available for the next ranking stage.

Metadata filters solve a different problem. They restrict candidates by facts such as tenant, region, product, or date. OpenAI's current Retrieval API applies attribute filters before semantic search. Filtering cannot rescue a relevant chunk that was parsed badly, but it can prevent an otherwise similar chunk from the wrong tenant or policy version from entering the candidate set.

Retrieval first, reranking second

Candidate retrieval is usually broad enough to favor recall. A reranker receives those candidates and reorders them for the query. Azure AI Search can apply semantic ranking to text and hybrid results when the search documents contain suitable text fields.

Reranking cannot recover a chunk absent from the candidate set. This ordering is therefore important:

query
  -> metadata and access filters
  -> keyword and vector candidates
  -> fusion
  -> semantic reranking
  -> relevance threshold and final top chunks
  -> generation

A relevance threshold trades coverage for precision. OpenAI's Retrieval API documents that raising score_threshold limits results to more relevant chunks but may exclude useful ones. Tune the threshold on answerable and unanswerable queries.

Manage the context as a budget

The context window is finite and measured in tokens. Google's Gemini token documentation defines it as the combined limit for input and output tokens, while specific limits depend on the model. The request must leave room for instructions, conversation history, retrieved evidence, the user's question, and generated output.

const availableEvidenceTokens =
  modelContextLimit
  - reservedOutputTokens
  - instructionTokens
  - conversationTokens
  - userQuestionTokens;

const selected = takeRankedChunksWithin(
  rerankedChunks,
  availableEvidenceTokens,
);

The function is application pseudocode, not a provider API. It makes the allocation visible and testable.

Wrong "If all documents fit, retrieval quality no longer matters."

Right A large context window changes what can fit. Google documents separate input and output token limits and says token counts contribute to billing. Neither fact establishes that every included passage is relevant. Retrieval remains a quality, latency, and cost choice.

Diagnose the stage that lost the evidence

One aggregate answer score hides several systems. Keep a trace containing the query, retrieved candidates, their scores, the reranked order, the final context, the response, and source references. Then label the failure at the earliest stage where the required evidence disappeared.

{
  "query": "Which policy covers ERR_AUTH_041?",
  "candidates": [{ "chunkId": "auth-9", "score": 0.71 }],
  "selectedContext": ["auth-9"],
  "response": "...",
  "sources": ["authentication-errors.md"]
}

The score has meaning only within the search system that produced it. Azure AI Search documents different score algorithms and ranges for full-text, vector, hybrid, and semantic ranking. Do not compare raw scores from separate ranking methods as though they shared one scale.

Measure retrieval and generation separately

For a labeled query set, Recall at K measures how much known relevant material appears in the first K results. Precision at K measures how many of those K results are relevant. Mean Reciprocal Rank focuses on the position of the first relevant result. Azure's RAG retrieval guidance recommends positive and negative examples and reports these metrics separately from the generated answer.

The final response needs its own checks. Microsoft Foundry's current RAG evaluator documentation separates retrieval quality, response relevance, response completeness, and groundedness. Its groundedness evaluator asks whether the response aligns with the supplied context without fabrication.

retrieval miss:
  required source is absent from top K

ranking miss:
  required source is present but too low to enter the final context

grounding miss:
  adequate context was supplied, but the response contains unsupported content

abstention success:
  the corpus lacks support, and the response declines to invent an answer

Wrong "A citation proves the sentence is supported."

Right Source references make evidence inspectable. Evaluate whether each claim aligns with that evidence. Microsoft defines groundedness as a separate response evaluation rather than treating a source reference as proof.

Tune candidate recall before final-context precision

Approximate nearest-neighbor search is a performance choice. Azure AI Search documents exhaustive KNN as a scan of the vector space and HNSW as approximate nearest-neighbor search. HNSW exposes configuration that trades index work, compute, latency, and recall. Treat those settings as service-specific controls and evaluate them on the deployed index.

A two-stage pipeline can retrieve more candidates than it sends to the generator. The first stage protects recall. Fusion, reranking, filters, and thresholds improve the smaller final set.

apply access and freshness filters during retrieval
  -> retrieve 40 candidates across keyword and vector search
  -> fuse and rerank
  -> pack the best supported passages into the evidence budget

The numbers are an experiment configuration, not a general recommendation. Azure's semantic ranker has provider-specific input limits, and other rerankers differ. Pin such values to the chosen service and API version.

If evidence is missing, tuning the generation prompt is downstream of the defect. Inspect chunk parsing, the embedding model used for documents and queries, filters, query formulation, candidate K, and search mode first. Azure's RAG guidance requires the same embedding model for stored chunks and vectorized queries.

Grounding needs an explicit no-evidence path

A threshold that excludes weak results may leave no usable context. The generation contract should represent that state directly.

if (selectedChunks.length === 0) {
  return {
    answer: "I could not find supporting information in the indexed sources.",
    sources: [],
  };
}

return generateFromSources({
  question,
  sources: selectedChunks,
  requireSourceReferences: true,
});

This application code makes abstention observable. Test it with negative queries whose answers are absent from the corpus. Azure's retrieval guidance expects negative examples to produce retrieval metrics near zero, while groundedness evaluation checks that generated content stays within the supplied context.

Freshness and authorization belong before generation as well. Attach filterable metadata such as tenant and policy date, apply those constraints during retrieval, and return source identity with every selected passage. A semantically similar chunk from the wrong tenant is still the wrong evidence.