AI and LLMs Senior interview questions and answers

75 questions for a AI and LLMs (Senior) interview - with correct answers, explanations and links to the docs.

Take this quiz
  1. 1.When is an LLM the wrong tool for a feature?
  2. 2.What is eval-driven development for AI features?
  3. 3.How should success criteria for an AI feature be defined?
  4. 4.What makes an eval dataset representative?
  5. 5.Why keep a held-out eval set you don't look at while tuning prompts?
  6. 6.Your new prompt scores 84% vs 82% on 50 cases. What's the right conclusion?
  7. 7.Why may offline eval scores and online metrics disagree?
  8. 8.What is benchmark contamination?
  9. 9.How can an LLM judge drift over time?
  10. 10.What is the cost-quality frontier when choosing models?
  11. 11.When does a batch API make sense for LLM workloads?
  12. 12.How do you keep a RAG index fresh as documents change?
  13. 13.What must happen in a vector store when a user exercises GDPR deletion?
  14. 14.When can a very long context replace RAG?
  15. 15.What is query rewriting in RAG?
  16. 16.What is agentic RAG?
  17. 17.When is fine-tuning worth it over prompting a large model?
  18. 18.Why pin model versions in production?
  19. 19.How should a team prepare for a model deprecation?
  20. 20.How can you reduce provider lock-in for LLM features?
  21. 21.What should tracing of an LLM feature capture?
  22. 22.An AI feature starts producing harmful output for many users. What is the first move?
  23. 23.What is the "lethal trifecta" for prompt injection in agents?
  24. 24.How can rendered Markdown leak data from an AI chat?
  25. 25.Why must code execution tools for agents run in a sandbox?
  26. 26.What should audit logs for an agent with real permissions record?
  27. 27.What is MCP tool poisoning?
  28. 28.How should authorization work for MCP tools that act on a user's data?
  29. 29.What does defence in depth mean for LLM safety?
  30. 30.Under the EU AI Act, what decides a system's obligations?
  31. 31.What are the four functions of the NIST AI Risk Management Framework?
  32. 32.How do you test an AI feature for unfair treatment of groups?
  33. 33.How should agents be evaluated beyond the final answer?
  34. 34.What does an end-state eval of an agent check?
  35. 35.What is a simulated user in chatbot testing?
  36. 36.What is shadow mode for a new AI model or prompt?
  37. 37.Why roll out prompt changes gradually behind flags?
  38. 38.What is a human review queue for AI outputs useful for?
  39. 39.How can a team measure the hallucination rate of a RAG assistant?
  40. 40.Why is abstention ("I don't know") an important behaviour to evaluate?
  41. 41.What should a team check about data retention when using an LLM API?
  42. 42.What is a risk of strict structured output constraints on complex reasoning tasks?
  43. 43.When is a reasoning model worth its extra cost and latency?
  44. 44.How do you estimate the ROI of an AI feature?
  45. 45.What is the main challenge of testing an AI coding agent?
  46. 46.How should ownership of prompts be organized in a product team?
  47. 47.What is the trade-off of letting an agent choose its own steps vs a fixed workflow at scale?
  48. 48.How do you test a guardrail without making it too strict?
  49. 49.What is a model card or system card used for?
  50. 50.How can retrieval leak data across tenants in a shared vector index?
  51. 51.What is a capacity risk specific to LLM features?
  52. 52.Why is copyright and licensing a concern for AI-generated output?
  53. 53.What is the role of guard tests when upgrading the model behind an agent?
  54. 54.How can you make LLM eval results reproducible?
  55. 55.Why should cost be part of an eval report?
  56. 56.What is an AI incident post-mortem expected to produce?
  57. 57.How should an agent handle content from tools as opposed to user instructions?
  58. 58.What's a sensible latency budget approach for a chat assistant with tools?
  59. 59.When does running a small model locally in the browser or device make sense?
  60. 60.How should a team decide which AI risks to address first?
  61. 61.Why are accessibility checks needed for AI chat interfaces?
  62. 62.What is the risk of letting a model generate and run database queries for analytics?
  63. 63.How do you test that an AI assistant respects user permissions in its answers?
  64. 64.What is the purpose of a red-team exercise before launching an AI feature?
  65. 65.How should an AI feature behave when its provider is down?
  66. 66.Why are long-horizon agents harder to evaluate than single calls?
  67. 67.What does "AI feature observability" add beyond normal APM?
  68. 68.How do you avoid an eval suite that only tests what the team already thought of?
  69. 69.What is a sensible policy for AI-generated content shown to users as fact?
  70. 70.Why can adding more tools to an agent make it worse?
  71. 71.What should be tested when an AI feature supports file uploads like PDFs?
  72. 72.Why is it risky to use the same model as both generator and only judge?
  73. 73.What is a practical first step to secure an existing LLM feature?
  74. 74.How should evaluation differ for a high-stakes domain like medicine or law?
  75. 75.What is a prompt regression test?