Context Windows — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— Context Windows
📋Count
⚙️Trim
🔄Chunk
✅Budget
🎯 Next: Test →
Step 1 / 4
Speed
Token Counting
tiktoken counts tokens before the API call — prevents truncation-surprises💡 Analogy
Like weighing luggage at home before the airport — know before you get there, not after.
Tiktoken is OpenAI's tokenizer — the same one used on the server. Count tokens before sending to predict cost and ensure you fit within the context limit.
Console Output
> count_tokens([{system: "You are..."}, {user: "Hello"}])
> system message: 4 + 12 = 16 tokens
> user message: 4 + 3 = 7 tokens
> Total: 25 tokens
●context_windows.py
Token CountingL6–13
1import tiktoken
2from openai import OpenAI
3 
4client = OpenAI()
5 
6# Count tokens BEFORE API call to avoid truncation
7def count_tokens(messages: list[dict], model: str = "gpt-4o") -> int:
8    enc = tiktoken.encoding_for_model(model)
9    total = 0
10    for msg in messages:
11        total += 4   # per-message overhead (role tokens)
12        total += len(enc.encode(msg["content"]))
13    total += 2       # reply primer tokens
14    return total
15 
16# Context management: sliding window for long conversations
17def trim_to_fit(messages: list[dict], max_tokens: int = 120_000,
18                model: str = "gpt-4o") -> list[dict]:
19    # Always keep system message (index 0)
20    system = [messages[0]] if messages[0]["role"] == "system" else []
21    conversation = messages[len(system):]
22 
23    # Remove oldest messages until within limit
24    while count_tokens(system + conversation, model) > max_tokens:
25        conversation = conversation[2:]  # remove oldest user+assistant pair
26 
27    return system + conversation
28 
29# Smart chunking for long documents
30def chunk_document(text: str, chunk_size: int = 2000,
31                   overlap: int = 200) -> list[str]:
32    enc = tiktoken.get_encoding("cl100k_base")
33    tokens = enc.encode(text)
34    chunks = []
35    start = 0
36    while start < len(tokens):
37        end = min(start + chunk_size, len(tokens))
38        chunks.append(enc.decode(tokens[start:end]))
39        start += chunk_size - overlap   # overlap for continuity
40    return chunks
41 
42text_tokens = count_tokens([{"role": "user", "content": "Hello, world!"}])
43print(f"Tokens: {text_tokens}")
43 lines
Timeline — 4 Steps
Token Counting
tiktoken counts tokens before the API call — prevents truncation-surprises💡 Analogy
Like weighing luggage at home before the airport — know before you get there, not after.
Tiktoken is OpenAI's tokenizer — the same one used on the server. Count tokens before sending to predict cost and ensure you fit within the context limit.
Console Output
> count_tokens([{system: "You are..."}, {user: "Hello"}])
> system message: 4 + 12 = 16 tokens
> user message: 4 + 3 = 7 tokens
> Total: 25 tokens

Was this page helpful?

Discussion

tiktoken counts tokens before the API call — prevents truncation-surprises

💡 Analogy

Like weighing luggage at home before the airport — know before you get there, not after.

Tiktoken is OpenAI's tokenizer — the same one used on the server. Count tokens before sending to predict cost and ensure you fit within the context limit.

Console Output

> count_tokens([{system: "You are..."}, {user: "Hello"}])
> system message: 4 + 12 = 16 tokens
> user message: 4 + 3 = 7 tokens
> Total: 25 tokens

tiktoken counts tokens before the API call — prevents truncation-surprises

💡 Analogy

Like weighing luggage at home before the airport — know before you get there, not after.

Tiktoken is OpenAI's tokenizer — the same one used on the server. Count tokens before sending to predict cost and ensure you fit within the context limit.

Console Output

> count_tokens([{system: "You are..."}, {user: "Hello"}])
> system message: 4 + 12 = 16 tokens
> user message: 4 + 3 = 7 tokens
> Total: 25 tokens