## Prompt edits deserve the same gate as a model swap

URL: https://www.evalshift.dev/blog/prompt-regression-testing
Published: 2026-08-13
Tag: prompts
Summary: A prompt edit changes behavior the way a model swap does. Where the prompt text lives decides whether CI diffs it or passes on silence.

Takeaways:
- A prompt edit is a behavior change with no diff to review — the same golden suite that gates a model swap gates it, unchanged.
- eval_config_hash covers version, prompts, defaults, evaluators and slices, and the baseline lookup skips every run whose hash differs.
- detection: manual puts the prompt body inside that hash, so every wording edit re-baselines the project and the PR check passes with no comparison at all.
- detection: python_string keeps the body in your application code — the hash holds still, the diff runs, and a regression reads as a regression.
- A permanently green EvalShift check usually means no comparable baseline was found, not that the suite passed.

A model swap gets a meeting. A prompt edit gets a commit. Both change what your product says to
users, and only one of them is reviewed like it matters — usually because a prompt diff is one
paragraph of English that reads fine and proves nothing.

The gate you already built for model migrations works on prompt edits unchanged: a golden suite, a
paired run, a diff against the base branch. But whether a prompt edit actually reaches that gate
depends on something most teams never look at — where the prompt text lives.

## The gate, briefly

On a pull request the EvalShift action runs the suite, uploads the run, asks the Cloud API for a
compatible baseline run on the base branch, fetches the server-side diff, updates one PR comment,
and sets the `evalshift/regression` commit status. Under `fail-on: regression` the check goes red
when the diff shows regressions.

The load-bearing word is *compatible*. The lookup scans recent `available` runs on the base branch
for the same suite, and skips every one whose `eval_config_hash` differs from the candidate's.
No compatible baseline means no diff — and no diff means the check passes.

## Where the prompt lives decides whether it is gated

`eval_config_hash` is a SHA-256 over the canonical JSON of your config snapshot: `version`,
`prompts`, `defaults`, `evaluators`, `slices`. The `prompts` block is in the hash. What is inside
that block depends on the detection mode:

```yaml
# detection: manual — the prompt body IS the config
prompts:
  - id: replay
    detection: manual
    content: "You are a support agent. Answer in at most three sentences.\n\n{input}"
    variables: [input]

# detection: python_string — the config only points at the body
prompts:
  - id: customer_routing
    detection: python_string
    path: prompts.py
    variable: AGENT_SYSTEM_PROMPT
    variables: [query]
```

With `detection: manual`, editing one word of the prompt changes `prompts[].content`, which changes
the config snapshot, which changes `eval_config_hash`. Your PR's run is now incomparable with every
run on `main`. The action finds no compatible baseline, reports exactly that in the comment, and
exits green.

With `detection: python_string`, `content` must be null — the config carries only `path` and
`variable`, and the body is AST-extracted from the Python file at run time. Edit the prompt and the
hash does not move: the baseline lookup matches, the server diffs the runs directly, and the
regression shows up as a regression.

> A permanently green EvalShift check usually means no comparable baseline exists, not that the
> suite is passing. If the check went green the same week someone rewrote the system prompt inline,
> that is the mechanism, not a coincidence.

The same rule governs everything else in the hash: retuning `defaults`, adding an evaluator, or
renaming a slice all re-baseline you on purpose. Prompt wording is the one that gets edited weekly
and the one nobody expects to sever the comparison.

## What still changes when the text changes

Keeping the prompt out of the config hash does not make edits invisible to the pipeline:

- The response cache is keyed on `{model, prompt, inputs, temperature, max_tokens[, history]}`, so
  an edited prompt is a cache miss. You cannot accidentally score new wording with old outputs.
- Every example is validated against every prompt — template variables covered — *before* any model
  call is dispatched. Dropping `{order_id}` from the template fails the run in seconds rather than
  after $9 of calls.
- `python_string` extraction never imports or executes your code. It AST-walks for a module-level
  string assignment and takes the last one; f-strings, concatenation, `.format()`, and function
  calls are rejected rather than evaluated.

That last rule has a sharp edge. A prompt assembled at runtime cannot be extracted, and the
documented workaround is to paste it into the config as `detection: manual` — which puts the body
back inside the hash and back into re-baselining on every edit. Prefer refactoring the template so
the literal is the literal and the variables are `{placeholders}` the suite fills in.

## Wire it once

Keep one copy of the prompt, in the application code that ships it, and point the config at it:

```yaml
prompts:
  - id: customer_routing
    detection: python_string
    path: app/prompts.py
    variable: AGENT_SYSTEM_PROMPT
    variables: [query]
```

Your app imports `AGENT_SYSTEM_PROMPT`; EvalShift reads the same file. There is no second copy to
drift, and a reviewer looking at the PR diff sees the prompt change and the eval result side by
side. Tool schemas follow the same no-second-copy rule by a different route: they live on the
suite examples, not in the config — `evalshift capture sync` records each example's toolset as a
content-addressed `toolset_ref` sidecar — so they never enter `eval_config_hash` either.

Then let the run answer the question the prose cannot. Both sides score paired per example, so
"the new prompt is more concise" becomes a delta per evaluator with a severity attached, and the
migration policy turns it into `pass`, `conditional_pass`, `fail`, or `inconclusive` without a
meeting. Roll out with `fail-on: never` for a week to collect baselines, then switch to
`fail-on: regression` once the comments match your judgement.

## The review question worth adopting

For every PR that touches a prompt: *did this run diff against a baseline, or against nothing?*
The comment answers it in one line. A prompt edit that produced no comparison has not been
tested — it has only been observed to compile.

## Keep reading

- [LLM regression testing in CI](/blog/llm-regression-testing-in-ci) — the gate itself, end to end.
- [How many eval cases do you need?](/blog/how-many-eval-cases-do-you-need) — sizing the suite that
  gate reads.
- [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls) — what changes in the trace when
  a system prompt changes.
- [Configuration](/docs/configuration) and [Baselines](/docs/baselines) — the config fields and the
  baseline model in full.