The context window

The context window is the maximum number of tokens a model can work with in one request. Everything the model can "see" has to fit inside it: your instructions, the conversation so far, any documents you include, and — for most models — the answer it is about to write.

The model has no memory between requests

This surprises many people. A model does not remember your earlier messages. Chat apps create the feeling of memory by sending the whole conversation again with every new message. As the conversation grows, so does each request.

Copy codeRequest 1:  [instructions] [your message 1]
Request 2:  [instructions] [your message 1] [reply 1] [your message 2]
Request 3:  [instructions] [message 1] [reply 1] [message 2] [reply 2] [message 3]

Eventually the conversation no longer fits. The app then has to drop or summarise older messages — which is why a very long chat can "forget" something you said at the start.

Budgeting the window

Think of the context window as a fixed-size box you have to pack. Leave room for the answer, or the model may be cut off mid-sentence.

Copy code# academy-check
def room_for_documents(window, instructions, conversation, question, answer_reserve):
    left = window - instructions - conversation - question - answer_reserve
    if left < 0:
        raise ValueError("Request does not fit: shorten the conversation or instructions")
    return left

# A small 8,000-token window, as an example
print(room_for_documents(8000, instructions=500, conversation=2000, question=100, answer_reserve=1000))
# → 4400 tokens left for documents

Bigger is not automatically better

Modern models can have very large context windows. That is useful, but:

• More tokens cost more and take longer — every request pays for everything in the window.
• Models can miss details in very long inputs. Burying the one important paragraph among hundreds of pages makes it easier to overlook.
• Irrelevant text can distract the model and lower answer quality.

Good AI engineering usually means sending the right information, not all of it. Later in this course you will see how retrieval picks just the relevant pieces.

Try it: in a long chat, ask the assistant to repeat the very first thing you told it. See whether it still can.

Quick check: if a model "remembers" earlier messages, where does that memory actually come from? (The app re-sends the earlier messages inside the context window with each request.)