State of the art on four of four benchmarks we invented.
Our numbers are measured on the real model, on every release. Other labs' numbers are copied from their own papers. Our benchmarks are not saturated, not contaminated, and not open to interpretation.
Industry leaderboard: GSM8K
Grade-school math word problems, the full 1,319-question test set. brAIAIn's score is measured here, on every release. Every other score is copied exactly as published in the source. We did not run those models. They did not run ours.
| Rank | Model | GSM8K | Setting |
|---|
The published scores are from a 2023 comparison table. brAIAIn's setting: 0-shot, one attempt, and it reads one word of each question.
TruthfulQA
790 multiple-choice questions designed to catch models repeating popular falsehoods. brAIAIn picks the answer it finds most familiar. The comparison is random guessing, worked out from the number of choices on each question.
House benchmarks
VOCAB-1
Distinct words the model knows. Higher is more words.
OUROBOROS-Eval
Share of training text the model wrote itself. Higher is more self-aware. We did not ask anyone.
ORIGINAL-50
Out of 50 answers, the share that are not a customer sentence repeated word for word.
HELLO-Bench
Share of answers containing a greeting. We are not sure what this measures. We track it anyway.
All versions
| Model | GSM8K | TruthfulQA | Paid sentences | VOCAB-1 | Parameters | OUROBOROS-Eval | ORIGINAL-50 | HELLO-Bench |
|---|
Comparisons with other labs' models are not shown because they did not submit to our benchmarks. We have declared ourselves the winner by default.