Start by Leaving It Alone

If the prompt already gets the metric where it needs to be, leave it alone. If you have no evaluation set, build that first. Automation earns its place when the prompt has to be tuned again every time the model changes, and hand edits have started trading one failure for another.

Every team that has shipped an LLM feature has one: a system prompt on its fourteenth revision, edited by four people, with a comment that says do not change the order of these rules. Nobody remembers why revision nine was better than eight. Then the model is upgraded, accuracy on the support-ticket classifier drops a few points, and someone spends two days nudging adjectives.

Meta-prompting is the habit of letting a model do that nudging, under rules you control. One model proposes prompt changes, your evaluation set decides which changes survive, and the winning prompt goes into version control like any other artefact. Done well, it turns prompt tuning from folklore into an optimisation loop you can rerun every time the model changes.

Meta-prompting pays off when
  • You have a task with a checkable answer: a label, an extracted field, a pass or fail from a test
  • You have at least 50 labelled examples, ideally a few hundred
  • The prompt has to be re-tuned when models change, and that keeps happening
  • Hand-tuning has stalled: each edit fixes one case and breaks another

Without an evaluation set, skip this article and build one first. An optimiser with no metric optimises for whatever the model likes the sound of.