The AI judge

What it sees

At a checkpoint the judge is given your expectation, the screenshot, the text taken from the page, the timing, and any file the run downloaded for it. It answers with a verdict, a confidence from 0 to 1, reasoning in sentences, and the evidence it relied on.

It is told to prefer what is observable over what the page claims, and to fail when the evidence cannot confirm the expectation rather than assume the rest went fine.

Disagreeing

On any verdict, Agree or Disagree. Disagreeing asks what it got wrong.

Those answers do three things:

1. They become a labelled set you can export.
2. They produce the agreement figure on the Trust page, split into missed failures and false alarms — the two mistakes cost very differently.
3. They teach the judge: your reason is turned into one short note that the judge reads every later time it judges that same checkpoint.

Learned corrections

Trust → Learned corrections lists every note, which checkpoint it belongs to, and when it was learned. You can rewrite one or remove it.

A note can only describe how to read the evidence. Anything that tries to decide the verdict on its own — "always pass" — is refused, whether it came from the model or from a person.

Learning is free: it does not spend AI credits.

Before we change the judge

When we change how the judge is prompted, we replay your labelled set against the new version and compare agreement with the previous one. Accuracy is measured, not asserted.