Open evaluation toolkit

Why I built it

Our evaluations lived in notebooks, so nobody could rerun anyone else's. The toolkit turns each eval into a test file that runs on every model change.

How it works

You write a task as a list of inputs and checks. The runner calls the model, scores the answers and writes a report with failing examples at the top.

What changed

Regressions now show up the night they happen instead of weeks later. Other teams adopted it without a migration guide because the files read like ordinary tests.