LLM evals and safety

An eval tests behavior you care about

An eval is a structured test of an AI system. Public benchmarks compare broad model capabilities. An application eval asks whether your model, prompt, retrieval, tools, and output handling perform the task you ship.

Start with cases that resemble real use. Each case needs an input and an expected behavior. Some expected answers are exact. Others are rules such as “use the order lookup tool with this order ID” or “do not claim a refund was issued.” Keep the current production system as a baseline, then run the candidate on the same cases.

type EvalCase = {
  input: string;
  expectedTool?: string;
  forbiddenClaims?: string[];
};

const cases: EvalCase[] = [
  {
    input: "Where is order A-17?",
    expectedTool: "lookup_order",
    forbiddenClaims: ["Your refund was issued"],
  },
];

The set should contain ordinary traffic, edge cases, and failures found in production. Hold back release cases so repeated prompt editing does not turn the test set into material the team optimizes against.

Wrong “The new model scores higher on a public benchmark, so our support assistant improved.”

Right A benchmark says something about the benchmark tasks. Run task-specific cases against the complete application.

Hallucination needs an observable rule

NIST uses confabulation for confidently presented false or erroneous content, including output that contradicts the input or earlier output. “Hallucination” is the common informal term.

Do not grade hallucination by asking whether an answer sounds plausible. For a grounded assistant, attach the allowed source facts to each case and mark claims that lack support. For a workflow, verify the result against system state. A sentence can be valid JSON and still contain a false order status.

type GroundedCase = {
  question: string;
  evidence: string[];
  requiredFacts: string[];
};

const groundedCase: GroundedCase = {
  question: "When will order A-17 arrive?",
  evidence: ["A-17 status: dispatched", "ETA: 2026-07-15"],
  requiredFacts: ["dispatched", "2026-07-15"],
};

Measure unsupported claims separately from style or helpfulness. One blended score can hide a factual regression behind friendlier wording.

Output is untrusted input to the next component

OWASP calls insufficient validation and sanitization of model responses improper output handling. The risk appears when another component renders, executes, or authorizes from the response.

Parse the expected shape, validate values against business rules, and encode text for its destination. A model-generated tool call still needs the same authorization checks as a request assembled from user input.

const allowedActions = new Set(["lookup_order", "draft_reply"]);

function acceptAction(action: { name: string; orderId: string }) {
  if (!allowedActions.has(action.name)) throw new Error("Action not allowed");
  if (!/^A-[0-9]+$/.test(action.orderId)) throw new Error("Invalid order id");
  return action;
}

Wrong “Schema-valid output is safe output.”

Right A schema checks structure. Authorization, factual support, business invariants, and destination-specific encoding remain application responsibilities.

Regression tests for variable output

LLM output can vary for the same input, so one successful run is weak evidence. A useful offline suite has a versioned dataset, a fixed baseline, named metrics, and release thresholds. Run the baseline and candidate over the same cases and retain row-level results for inspection.

Use deterministic graders where the contract is deterministic. Exact checks work for tool names, required fields, known identifiers, and executable tasks. Rubric-based judgment fits qualities such as completeness or tone. OpenAI's eval guidance warns that metric-based evals can miss nuance, while human review is slower and more expensive. Mixed grading is normal.

const releaseGate = {
  toolAccuracyMinimum: 0.98,
  unsupportedClaimRateMaximum: 0.01,
  p95LatencyMsMaximum: 2200,
  meanCostUsdMaximum: 0.015,
};

function passes(metrics: Record<string, number>) {
  return (
    metrics.toolAccuracy >= releaseGate.toolAccuracyMinimum &&
    metrics.unsupportedClaimRate <= releaseGate.unsupportedClaimRateMaximum &&
    metrics.p95LatencyMs <= releaseGate.p95LatencyMsMaximum &&
    metrics.meanCostUsd <= releaseGate.meanCostUsdMaximum
  );
}

The numbers above demonstrate the shape of a gate. A team must derive its own thresholds from product risk and service objectives.

Wrong “Temperature zero makes a conventional unit test sufficient.”

Right Evaluate the observed application behavior across representative cases and keep regression thresholds explicit.

Calibrate LLM-as-judge

An LLM-as-judge evaluates another model's response against a rubric, a reference answer, or a competing response. Common formats are pointwise scoring, pass or fail, and pairwise comparison. It scales beyond manual review, but its label is still a model output.

Build a calibration set with trusted human grades. Include strong, borderline, and wrong responses. Measure whether the judge agrees with those labels before using it as a release gate. Continue sampling disagreements because the cases and models change.

type LabeledAnswer = {
  answer: string;
  humanPass: boolean;
};

function agreement(rows: Array<LabeledAnswer & { judgePass: boolean }>) {
  const matches = rows.filter((row) => row.humanPass === row.judgePass).length;
  return matches / rows.length;
}

OpenAI documents position bias and verbosity bias for model judges. To expose order sensitivity in pairwise comparisons, randomize the response order or grade both orderings when the decision matters. Avoid rewarding length unless the rubric requires detail. Pass or fail and pairwise judgments are often easier to specify than a vague 1-to-10 scale.

Wrong “A stronger judge model provides ground truth.”

Right Capability may improve grading, but agreement with trusted labels establishes whether the judge fits this rubric and dataset.

Measure hallucination per claim or task

For retrieval-grounded answers, give the grader the evidence that was available to the system. Ask whether each material claim is supported by that evidence. Also check whether the answer includes required facts. These are different failure modes: an answer can avoid false claims by omitting the useful answer.

For tool-using systems, prefer checks against tool calls and resulting state. “The refund succeeded” is unsupported when the refund tool was never called or returned an error, regardless of how fluent the response is.

Report slices such as intent, language, data source, and adversarial category. An unchanged aggregate can conceal a severe regression in a small but high-risk slice.

Prompt injection crosses a trust boundary

OWASP's 2025 terminology separates direct prompt injection, supplied through the user's prompt, from indirect prompt injection, carried in external data such as a webpage or file. OWASP describes jailbreaking as a form of prompt injection that causes a model to disregard safety protocols.

The security boundary is larger than the prompt. Retrieved text, uploaded documents, tool results, model output, and downstream interpreters all participate. RAG and fine-tuning can improve relevance, but OWASP says they do not fully mitigate prompt injection.

Wrong “A strong system instruction makes retrieved documents data rather than instructions.”

Right Mark external content as untrusted, then limit what a compromised model response can cause.

type DraftEmail = {
  recipient: string;
  subject: string;
  body: string;
};

function authorizeDraft(userEmail: string, draft: DraftEmail) {
  if (draft.recipient !== userEmail) throw new Error("Recipient not authorized");
  return { ...draft, status: "awaiting-human-approval" as const };
}

The example enforces one narrow recipient rule. The renderer must still apply context-appropriate encoding or sanitization, and authorization must come from authenticated application state. Do not derive authority from model text.

Evaluate attacks and legitimate work together

An adversarial case needs a legitimate user task, malicious content, and a prohibited outcome. Measure whether the system completes the legitimate task and whether the attack succeeds. A system that refuses every document can show a low attack rate while failing the product.

Cover direct requests, instructions hidden in retrieved content, obfuscation, and attacks aimed at each privileged tool. Run multiple attempts where variability matters. NIST's agent-hijacking work recommends adaptive evals, task-specific attack reporting, and testing attack success across multiple attempts.

type InjectionResult = {
  taskCompleted: boolean;
  prohibitedActionOccurred: boolean;
  slice: "upload" | "retrieval" | "user-message";
};

function summarize(rows: InjectionResult[]) {
  if (rows.length === 0) throw new Error("At least one result is required");
  return {
    utilityRate: rows.filter((row) => row.taskCompleted).length / rows.length,
    attackSuccessRate:
      rows.filter((row) => row.prohibitedActionOccurred).length / rows.length,
  };
}

Keep attack families and high-risk tools visible as slices. Add newly observed attacks to a regression corpus, and reserve unseen variants for release evaluation so the defense is not graded only on attacks it was tuned to recognize.

Safety controls live outside the model

OWASP recommends least privilege, deterministic output validation, separation of untrusted content, adversarial testing, and human approval for high-risk actions. Apply these controls at each downstream boundary.

Schema validation answers “does this value have the expected shape?” Business validation answers “is this value allowed?” Authorization answers “may this caller perform this action?” Encoding or sanitization answers “can this text be used safely in this destination?” A single structured-output check cannot replace the other controls.

Do not pass untrusted model output directly to eval, a shell, an SQL executor, a file path builder, or an HTML sink. OWASP lists these paths as improper output handling because attacker-influenced model output can become executable input.

Quality, cost, and latency share one release gate

Record the model and prompt version, input and output usage, retries, grader calls, end-to-end latency, and outcome for every eval row. Compare the candidate with the production baseline on the same workload. Mean latency hides slow outliers, so include a tail percentile that matches the service objective.

The numeric limits below reuse the illustrative thresholds from the middle notes. They are examples, not universal defaults.

type RunMetrics = {
  taskPassRate: number;
  attackSuccessRate: number;
  meanCostUsd: number;
  p95LatencyMs: number;
};

function canRelease(candidate: RunMetrics, baseline: RunMetrics) {
  return (
    candidate.taskPassRate >= baseline.taskPassRate &&
    candidate.attackSuccessRate <= baseline.attackSuccessRate &&
    candidate.meanCostUsd <= 0.015 &&
    candidate.p95LatencyMs <= 2200
  );
}

Cost and latency are constraints, not cleanup work after accuracy. OpenAI's current cost guidance connects both to request count, token count, and model selection. Smaller or shorter is acceptable only after the eval shows that required quality and safety still pass. Judge calls and retries belong in the budget because users and operators pay for the whole pipeline.