AI and LLMs · Junior · Question 3 of 75
Why does the same text in Ukrainian often use more tokens than in English?
- 1Models translate non-English text to English before counting
- 2Tokenizers favour English, so other languages split into more pieces
- 3Ukrainian text is always sent to the model twice for accuracy
Show the answer
Correct answer: Tokenizers favour English, so other languages split into more pieces
Explanation
More tokens mean higher cost, slower responses and less room in the context window for the same content.