What is the context window?

AI models don’t read words; instead they read chunks of characters called tokens. Typically 1 token is roughly equivalent to 0.75 words (in English), so 1,000 tokens will represent around 750 words. So when a model has a context window of 8,000 tokens, for example, it means it can handle roughly 6,000 words of data at once.

Each context window needs to hold three things simultaneously: the rules telling the AI how to behave; the history of the current chat; and the space required for the AI to generate its next answer.

If the limit is 8,000 tokens and your conversation history is 7,900 tokens long, the AI only has 100 tokens left. If a conversation exceeds the context window, the model might start deleting the oldest parts of the conversation.

The context window is linked to the computational resources available to the model. If you increase the context window length by 2x, the power required increases by roughly 4x. So models with larger windows are much more expensive to run.

Sometimes even if a model can accept a lakh tokens, it may struggle to find a piece of information buried in the middle. This is called the ‘lost in the middle’ phenomenon.