Context Window
The context window is the maximum number of tokens an LLM can process in a single execution step. This limit includes both the input prompt tokens and the gener...
Definition
The context window is the maximum number of tokens an LLM can process in a single execution step. This limit includes both the input prompt tokens and the generated output tokens.
Why It Exists
Under the hood, standard self-attention mechanisms scale quadratically () in memory usage relative to sequence length. A model with an infinite context window would require infinite GPU memory. Therefore, models must enforce a strict context limit to guarantee stable hardware execution.
How It Works
SLIDING CONTEXT WINDOW
Full Conversation History (12,000 tokens)
┌──────────────────────┬──────────────────────┬──────────────────────┐
│ Round 1 Chat Log │ Round 2 Chat Log │ Latest Prompt │
└──────────────────────┴──────────────────────┴──────────────────────┘
◄── [PRUNED / DROPPED] ──► ◄─── [ RETAINED IN ACTIVE CONTEXT ] ───►
(Context limit threshold: 8,000 tokens)
1. **Attention Matrix Limit**: During inference, the GPU allocates memory for the attention matrix, which tracks relationships between all input and output tokens.
2. **Buffer Limits**: If a request exceeds this window size, the server rejects it with a context length error.
3. **Sliding Window Pattern**: To maintain conversation history, the application layer must prune, summarize, or discard older messages once they exceed the context limit.Real-World Example
Gemini 1.5 Pro has a context window of 2 million tokens. This allows you to upload entire codebases, textbooks, or hours of video context directly in a single prompt. In contrast, older models (like GPT-3.5 with a 4k window) require aggressive pruning or RAG setups.
Python Example
Here is a sliding window context manager that prunes conversational history to fit within a target token limit:
from dataclasses import dataclass
from typing import List, Dict
@dataclass
class ChatMessage:
role: str
content: str
token_cost: int
class ConversationBuffer:
"""Manages chat history, pruning oldest messages to fit within a context window."""
def __init__(self, max_context_tokens: int):
self.max_tokens = max_context_tokens
self.history: List[ChatMessage] = []
def add_message(self, role: str, content: str, estimated_tokens: int):
self.history.append(ChatMessage(role, content, estimated_tokens))
self._prune_context()
def _prune_context(self):
# Keep pruning the oldest message until the total fits the window
while self.total_tokens() > self.max_tokens and len(self.history) > 1:
removed = self.history.pop(0)
print(f"Pruned message from history: '{removed.content[:20]}...' (Saved {removed.token_cost} tokens)")
def total_tokens(self) -> int:
return sum(msg.token_cost for msg in self.history)
def get_payload(self) -> List[Dict[str, str]]:
return [{"role": msg.role, "content": msg.content} for msg in self.history]
if __name__ == "__main__":
# Context window limit of 100 tokens
buffer = ConversationBuffer(max_context_tokens=100)
buffer.add_message("user", "Hello, how do I setup my server environment?", 30)
buffer.add_message("assistant", "Install Python, then activate your virtual environment.", 40)
buffer.add_message("user", "What is the command to activate on Windows?", 25)
# Adding this message exceeds the 100-token limit, triggering pruning
buffer.add_message("assistant", "Run '.\\venv\\Scripts\\Activate.ps1' in PowerShell.", 35)
print("Final Active History:", buffer.get_payload())Interview Questions
- Beginner: What happens when an API call exceeds the context window limit?
- Intermediate: What is the difference between input tokens and output tokens in relation to the context window?
- Advanced: Why does attention mechanism complexity scale quadratically with context length, and what architectures solve this?
Interview Answers
- Beginner: The API call will fail, returning an error response (commonly HTTP status 400 or 422) stating that the prompt length exceeds the maximum allowed token limit.
- Intermediate: Input tokens are the tokens in your prompt, including system instructions and context. Output tokens are the tokens generated by the model. The sum of input and output tokens must remain below the context window limit. Additionally, providers often place separate limits on output tokens.
- Advanced: Standard self-attention compares every token in a sequence with every other token, requiring calculations. This makes the computational complexity and memory usage scale quadratically (). Modern architectures solve this using linear attention models, FlashAttention (which optimizes GPU SRAM memory layout), or state-space models (like Mamba).
Common Mistakes
- Ignoring output buffer: Allocating your entire context window to the input prompt, leaving zero tokens for the model to generate a response. Always reserve a buffer for the expected output tokens.
Assignment
Write a function that calculates if a list of messages fits within a target limit. If it does not, return a summarized instruction context instead of the raw history.