Latency Analysis — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— LLM Latency Analysis
📋Timer
⚙️Stream
🔄TTFT
✅Calculate
🎯 Next: Test →
Step 1 / 4
Speed
Start Timer
Record t_start before the API call — measures full round-trip💡 Analogy
Like a race starter pistol — record exact start time before anything moves.
Use time.perf_counter() for high-resolution timing. Start before the API call to capture network + processing latency.
Console Output
> t_start = 1716574800.123456 (perf_counter)
> t_first_token = None
> tokens = 0
●latency_analysis.py
Start TimerL13–16
1import time
2import openai
3from dataclasses import dataclass
4 
5@dataclass
6class LatencyMetrics:
7    ttft_ms: float          # Time to first token
8    tps: float              # Tokens per second
9    total_latency_ms: float
10    total_tokens: int
11 
12def measure_streaming_latency(prompt: str) -> LatencyMetrics:
13    client = openai.OpenAI()
14    t_start = time.perf_counter()
15    t_first_token = None
16    tokens = 0
17 
18    stream = client.chat.completions.create(
19        model="gpt-4o-mini",
20        messages=[{"role": "user", "content": prompt}],
21        stream=True,
22    )
23 
24    for chunk in stream:
25        delta = chunk.choices[0].delta.content
26        if delta:
27            if t_first_token is None:
28                t_first_token = time.perf_counter()  # TTFT measured here
29            tokens += 1
30 
31    t_end = time.perf_counter()
32    ttft = (t_first_token - t_start) * 1000
33    total = (t_end - t_start) * 1000
34    generation_time = (t_end - t_first_token) * 1000
35 
36    return LatencyMetrics(
37        ttft_ms=ttft,
38        tps=tokens / (generation_time / 1000),
39        total_latency_ms=total,
40        total_tokens=tokens,
41    )
42 
43metrics = measure_streaming_latency("Explain transformers in 3 sentences.")
44print(f"TTFT: {metrics.ttft_ms:.0f}ms | TPS: {metrics.tps:.1f} | Total: {metrics.total_latency_ms:.0f}ms")
44 lines
Timeline — 4 Steps
Start Timer
Record t_start before the API call — measures full round-trip💡 Analogy
Like a race starter pistol — record exact start time before anything moves.
Use time.perf_counter() for high-resolution timing. Start before the API call to capture network + processing latency.
Console Output
> t_start = 1716574800.123456 (perf_counter)
> t_first_token = None
> tokens = 0

Was this page helpful?

Discussion

Record t_start before the API call — measures full round-trip

💡 Analogy

Like a race starter pistol — record exact start time before anything moves.

Use time.perf_counter() for high-resolution timing. Start before the API call to capture network + processing latency.

Console Output

> t_start = 1716574800.123456 (perf_counter)
> t_first_token = None
> tokens = 0

Record t_start before the API call — measures full round-trip

💡 Analogy

Like a race starter pistol — record exact start time before anything moves.

Use time.perf_counter() for high-resolution timing. Start before the API call to capture network + processing latency.

Console Output

> t_start = 1716574800.123456 (perf_counter)
> t_first_token = None
> tokens = 0