AI and LLMs · Junior · Question 2 of 75
What is a token in the context of LLMs?
- 1A chunk of text, often part of a word, that the model processes
- 2A single complete sentence from the user's message
- 3A password that lets an application call the model's API
Show the answer
Correct answer: A chunk of text, often part of a word, that the model processes
Explanation
Text is split into tokens by a tokenizer. In English one token is roughly three quarters of a word; other languages and code often need more tokens per word.