Research · Scaling
The road to AGI.
Every frontier lab publishes a scaling plan. This is ours. The model learns one paid sentence at a time, so the plan is mostly arithmetic. Every figure below is computed from the live model when this page loads.
Words written by people-
Words per paid sentence-
Current pace-
Next milestone-
Training text, to scale
Logarithmic. Each gridline is ten times more text than the one below it. The purple bar is brAIAIn.
Milestones
What it would take to get there, at $5 a sentence and the current pace.
Loading
How we computed this
- Other labs report training data in tokens. We convert at about 0.75 words per token, the rule of thumb OpenAI gives.
- Only text written by people counts. Each version also trains on text the previous version wrote. That is excluded here, over the model's objections.
- Pace is paid sentences since the first sale, divided by the days since then. Projections assume the pace and the average sentence length both hold. They will not.
- Real models also depend on data quality, compute and architecture. We have none of those. We have the arithmetic.
Sources: GPT-3 paper (300 billion tokens) · Chinchilla paper (1.4 trillion tokens) · Llama 2 paper (2 trillion tokens) · Llama 3 paper (over 15 trillion tokens) · BERT paper (3.3 billion words) · OpenAI on tokens · CDC milestones at 30 months · Folger Shakespeare Library