The context window is the maximum amount of information that a language model can process simultaneously, including both the user input and the response it generates. It is measured in tokens, the minimum units into which the model breaks down text (one token is approximately equivalent to three-quarters of a word in English). For example, a 128,000-token window allows handling the equivalent of a short book in a single interaction.
Its size matters because it determines how much the model "remembers" within a conversation or document. If the exchange exceeds the limit, the oldest content is discarded or truncated, which can cause the model to "forget" instructions or details given at the beginning.
In practice, it is worth keeping in mind that:
- A large window does not guarantee that the model will use all the information well; it tends to pay more attention to the beginning and the end.
- Processing more tokens increases the cost and latency of each query.
Therefore, it is advisable to summarize or structure the context instead of saturating it.