Q. Consider the following statements regarding the behavior of AI Tokens in Large Language Models (LLMs):
1.A token represents the smallest unit of data processed by an LLM, and it can consist of a single character, a part of a word, or an entire word.
2.The context window of an AI model represents a flexible, unlimited storage space that allows the model to remember all historical interactions indefinitely without re-sending data.
Which of the statements given above is/are correct?
Answer: A
Notes:
Detailed Explanation
- Statement 1 is correct: LLMs do not read text word-for-word like humans; instead, tokenizers break text down into tokens. These tokens are then converted into numerical vectors that the model processes using mathematical patterns.
- Statement 2 is incorrect: Every AI model has a finite context window limit (e.g., 128k tokens), which strictly defines how much data it can process at one time. Furthermore, LLMs are generally stateless, meaning they process data in chunks and do not retain memory of who a user is unless the previous tokens of the conversation are explicitly re-sent to the model in each new prompt.
