AI and LLMs · Middle · Question 75 of 75
Why run your own evals on a new model before switching to it?
- 1Benchmarks don't show how it does on your tasks and formats
- 2Providers forbid switching models without an approval
- 3New models always produce worse answers than older ones
Show the answer
Correct answer: Benchmarks don't show how it does on your tasks and formats
Explanation
A model that's better on public leaderboards may break your JSON format or tone. Your eval suite is the real acceptance test.