Every AI-graded product faces the same objection, and it's a fair one: who grades the grader? If a model tells you your answer is "senior level", why should you believe it any more than a horoscope?
We think the only honest response is to show the machinery. This post documents how Gapmap grades free-form answers. Not the marketing version, the actual mechanics.
The rubric is the brain, the model is the executor
The model never decides what a good answer is. That decision is made ahead of time, by humans, in a rubric that ships with every question:
- Criteria with weights. Each question has explicit criteria that sum to 1. "Explains why the closure captures the variable, not its value" might carry weight 0.35; a side detail carries 0.1. Your score is the weighted sum of what you covered.
- Senior signals. Above-and-beyond markers: memory implications, edge cases, production trade-offs. These push a strong answer into the top band.
- Red flags. Fundamental misconceptions. Hitting one doesn't just cost points, it caps your score below the band you'd otherwise get. If you fluently describe the event loop but claim JavaScript runs callbacks in parallel threads, no amount of fluency keeps you in "senior".
The grading prompt enforces one more rule that matters in practice: naming a term earns nothing. A criterion counts only if the answer explains it. Dropping "microtask queue" without saying what it changes is graded exactly like not knowing it. This is the single biggest gap between our grading and a keyword-matching quiz.
Bands, and what happens at their edges
Scores map to interview bands: junior, middle, middle+, senior. The interesting part is the boundaries. A model grader has run-to-run variance; the same answer can score 63 one run and 68 the next. In the middle of a band that's noise. At a boundary it flips your verdict.
So near a band edge the grader earns its verdict differently: it asks a follow-up question targeting your weakest criterion, which is what a real interviewer does when they're not sure about you. Your follow-up answer is graded together with the original in one pass, and the combined transcript decides which side of the boundary you land on. The follow-up can move the score in either direction. Bluffing gets more expensive; genuine understanding gets a second chance to show itself.
Golden answers: the grader takes its own exam
Every question ships with calibration answers written in advance: a genuinely senior answer, a solid-but-incomplete middle one, a confidently wrong one, a keyword-stuffed one. Each has an expected score range.
Before any change to prompts, rubrics, or models goes live, the whole calibration suite runs. If the grader scores the keyword-stuffed answer as senior, or drops the real senior answer to middle, the change does not ship. The eval suite is wired into the build itself, so content mistakes fail the build instead of a user's session.
Writing these calibration answers taught us more about grading than anything else. Early on, our "middle" answers kept scoring too high because they were well-written hedges. The fix wasn't prompt magic. We had to accept that a middle-level answer doesn't hedge, it misses whole criteria, and rewrite the calibration set to encode that.
Every verdict is versioned
Each stored verdict carries the grader version that produced it. When we improve the prompts or rubrics, old scores stay interpretable: we know exactly which brain graded what. And when a user disputes a verdict (that flow is coming), interesting disputes get promoted into the calibration set. Real users continuously harden the grader.
What this buys you
Honesty, mostly. The system can still be wrong; any grader can, human interviewers included. But it's wrong in auditable ways: every score decomposes into criteria covered, signals shown, and flags hit, and you see that decomposition instead of a bare number. When it says you're middle in this-binding, it tells you which two criteria separate you from middle+.
We think that's the standard AI grading owes you. If you want to see it applied to your own knowledge, the diagnostic is free.