Prompt Optimization Loop
From a one-off rewrite to refinement loops on failures, candidate search and compiled DSPy programs. How to optimise prompts against a metric without overfitting, and how to review what the optimiser produces.
Meta-prompting uses an LLM to generate, evaluate, and improve prompts. Instead of manually writing prompts, you describe the task and let the model generate a better prompt. DSPy (Declarative Self-improving Python) automates prompt optimisation — you write the program structure, DSPy optimises the prompts using your evaluation set. Automatic Prompt Optimisation (APO) iterates: generate prompt → evaluate on test set → identify failures → improve prompt → repeat.
The pipeline
Each stage in order — click any step to read what it does.
Interactive Pipeline Flow
Architecture diagrams
APO iterative loop for automated prompt improvement.
Key design decisions
The trade-offs worth knowing before you build this.
When your prompt performs poorly, ask GPT-6.1 Sol to critique it given the failure examples. 'Given these misclassifications, what's unclear or missing in my prompt?' often produces insights faster than manual analysis.
With DSPy, you write the program structure (retrieve, generate, score) and DSPy generates the optimal prompts automatically against your eval set. Code changes without prompt rewrites.
Meta-prompting's power comes from automated evaluation. Without a labeled dataset to measure prompt quality objectively, you're optimising in the dark. Build the eval set first, optimize second.
“Improve the triage prompt without moving to a bigger model.”
The same request as in Read, with the values it has at each stage. Step through it, or click any stage. Values are illustrative for this scenario.
Baseline
The current prompt scores 82% on 150 labelled tickets, split into 100 for tuning and 50 held out
Diagnose
GPT-6.1 Sol reads the 18 failures and the prompt, and summarises: “billing vs refund confusion; urgent tickets under-flagged”
Propose
It writes 4 candidate prompts, each targeting the diagnosed failures in a different way
Score
Each candidate runs on gpt-6-luna over the 100 tuning tickets. The best scores 89%.
Iterate
Three rounds later the best candidate scores 91% on tuning tickets
Hold-out
On the 50 untouched tickets it scores 88%, against 81% for the original. It ships.
The same request, failing at “Iterate”
After eight rounds the prompt hit 96% on the tuning tickets and 83% on the held-out set. It had memorised quirks of those 100 tickets: phrases copied from them, special cases for one-off wording. The hold-out set is what exposes overfitting, so it must never be shown to the optimiser.
Continue reading
All Prompt Engineering articles →Red Teaming LLM Applications: Attack It Before Your Users Do
Threat model first, then manual sessions, then automated scans with garak, promptfoo and PyRIT. How to find jailbreaks, data leaks and unsafe tool calls, fix them in the architecture, and keep them fixed in CI.
Tree-of-Thought: Search Over Reasoning Paths
Propose several next steps, judge the partial states, keep the best and expand them. How to build a Tree-of-Thought search, why the evaluator decides everything, and when a reasoning model makes it unnecessary.
Constrained Generation: Beyond Hope-Based Parsing
Enforce output structure at the token level with grammars and token masks — how it differs from native JSON mode, and when the strictness costs you reasoning quality.
Discussion
Sign in to share your feedback and join the discussion.