LLM-as-Judge

LLM-as-Judge is scoring an output with a second model call against a written rubric. It exists for outputs with no single correct text to compare against, a summary, an answer, a rewritten email, where a test-suite assertion has nothing to match. The rubric states what a good answer must contain and what disqualifies it; the judge receives the input, the output, and the rubric, and returns a score with a reason.

It works because judging is a smaller job than producing. Noticing that a summary dropped the deadline is easier than writing the summary, and the judge is only asked to notice.

Two things decide whether the score means anything. The first is the rubric: a vague rubric produces a confident score that measures nothing, so writing it is most of the eval. The second is who judges. A generator reviews its own output too generously, so the judge is a separate call from the one that produced the work. Family matters too: a model grading output from its own family shares the tendencies that produced it and will approve the fluent, plausible, wrong answer it would have written itself. Where the call matters, route the judge to a different vendor’s model, and spot-check a sample of judged cases by hand rather than trusting the aggregate.