AI and LLMs · Middle · Question 44 of 75
What is quantization of a model?
- 1Counting how many tokens a model can generate per second
- 2Splitting a model into several smaller models by topic
- 3Storing weights with fewer bits to save memory and speed up inference
Show the answer
Correct answer: Storing weights with fewer bits to save memory and speed up inference
Explanation
8-bit or 4-bit weights fit larger models on smaller GPUs, often with a small quality loss that evals should measure.