AI and LLMs · Middle · Question 26 of 75
How should an LLM eval suite be used in CI?
- 1Run it only in production on real users
- 2Run it once before the launch and then never again
- 3Run it on prompt or model changes and fail on score drops
Show the answer
Correct answer: Run it on prompt or model changes and fail on score drops
Explanation
Thresholds per metric make prompt edits safe, like unit tests for code. Keep a fast subset for every PR and a full run nightly.