Can you compare the three sources?
Prose and code
Content that comes from a person, a model, or a machine. All of it is treated as untrusted: no raw HTML, no remote images, and no link the reader cannot see the destination of. Markdown that is still arriving stays stable rather than reflowing on every character.
Findings
The measurements agree with two of the three sources.
- Source A matches within 2%
- Source B uses a different unit
ratio = observed / expected
| Source | Agrees |
|---|---|
| A | yes |
| B | after conversion |
Running the tests now:
pnpm test
ratio = observed / expected
Source Agrees A yes B after conversion Set temperature to 0.0 for a deterministic answer, or cap the reply with max_tokens.
export const config = {
models: ["gpt-4o", "claude-3.5-sonnet"],
benchmarks: ["mmlu", "humaneval"],
maxConcurrency: 4,
};$ evals run --model gpt-4o --bench mmlu
Evaluating 14,042 questions…
Complete. Score 87.4 / 100
bode open 0190a5d0-0000-7000-8000-000000000001