LLM capabilities are not proportional to size: they emerge suddenly at certain scales. A 1B parameter model can't reason about math or write code. A 70B model can do both. Scaling laws predict these emergence points and guide compute allocation.
Scaling Laws
Loss improves predictably with model size and compute.
L(N) = a * N^(-α)
where: N = number of parameters α ≈ 0.07 (empirically observed) loss decreases ~7% per 10× parameter increase
Compute-Optimal Training
Given a budget, how many parameters and training tokens?
Chinchilla scaling: N ≈ C / (6L)
where C is total compute, L is tokens
Implication: Double model size → double training tokens Not: Train for longer with same size
Emergent Abilities
Certain capabilities emerge suddenly above threshold sizes.
- 1B parameters: No in-context learning
- 10B parameters: Basic in-context learning
- 70B parameters: Complex reasoning, code generation
- 500B+: Long-context understanding, multi-step planning
# Small model: Struggles with math reasoning
# Large model: Solves multi-step math problems # Why? Capability threshold at ~30B+ parameters
Conclusion
Scaling laws guide research and deployment. Understanding emergence explains why larger models behave qualitatively differently. Chinchilla-optimal scaling balances model size and training data. This completes the LLM fundamentals series. Next: advanced topics like mixture-of-experts and retrieval augmentation.