AI and LLMs · Senior · Question 7 of 75
Why may offline eval scores and online metrics disagree?
- 1Offline evals use a different model than production
- 2Online metrics are always calculated incorrectly
- 3Eval data or criteria don't match real usage
Show the answer
Correct answer: Eval data or criteria don't match real usage
Explanation
When they diverge, update the eval set with real traffic patterns and revisit what the metric actually rewards.