https://www.evalshift.dev/blog Field notes on migrating LLM models safely: golden suites, paired evaluation, regression gates in CI, and evaluator reliability. en https://www.evalshift.dev/blog/what-a-passing-migration-proves https://www.evalshift.dev/blog/what-a-passing-migration-proves Wed, 16 Sep 2026 00:00:00 GMT An EvalShift migration report that said PASS: a 71% cheaper model inside every policy budget. Where each threshold came from, and what PASS does not claim. https://www.evalshift.dev/blog/cheaper-faster-and-still-a-fail https://www.evalshift.dev/blog/cheaper-faster-and-still-a-fail Sat, 12 Sep 2026 00:00:00 GMT A candidate model cut cost 59% and latency 75% with zero failed calls. The migration report still said FAIL, because the agent's behavior changed. https://www.evalshift.dev/blog/what-breaks-when-you-switch-llms https://www.evalshift.dev/blog/what-breaks-when-you-switch-llms Thu, 20 Aug 2026 00:00:00 GMT A model swap is a behavior change with no diff to review. What moves besides the answer text, and why the worst regressions read as correct. Your refund has been processed. Every text-based check passes. The sentence is fluent, on topic, and exactly what the old model said. The refund did not happen. If you are scoring outputs, this regression is not merely hard to catch — it is invisible by construction, because the output is correct and the behavior is not. The same shape covers most of the expensive failures: the `verify_payment` call that stops firing, the retry loop that starts, the confirmation step that moves after the write. What changed is the trace. The prose stayed still. ## You are usually changing two things at once Prompts are coupled to models. The system prompt in production has been tuned — often over months, often by accretion — against one model's quirks. Point it at a different model and some of that tuning becomes dead weight and some becomes actively harmful. So the honest migration usually involves editing the prompt too, and now the comparison has two independent variables in it. When quality moves, nobody can say whether the model did it or the rewrite did. The fix is boring: change one thing per run. Baseline old model with old prompt. Run new model with old prompt — that is the model's effect, unflattering as it may be. Then tune the prompt for the new model and run again, against the same frozen cases. Two comparisons, each interpretable, instead of one that isn't. ## Hand-written test cases test the paths you already handle Writing eval prompts by hand feels productive and produces a suite shaped like your mental model of the product. That is the problem. The prompts you can think of are the interactions you already understand well enough to have handled. Real failures come from the inputs you would never have written down: the eleven-turn conversation where a constraint set in turn three quietly expires, the message that arrives with half the context missing, the turn right after a tool returned an error, the customer typing in a language your template never anticipated. You cannot reconstruct those from memory. You have to record them. Which gives a workflow, independent of what you use to run it: ```text real agent behavior ↓ capture representative cases ↓ freeze a golden suite ↓ baseline model vs candidate model, same inputs ↓ compare behavior, not just text ↓ ship or reject ``` The freezing step is the one people skip. If cases are still being edited while models are being compared, two things are moving and the diff between them describes neither. ## An average is not a verdict Model A scores 0.91, model B scores 0.89, and someone screenshots it into the migration thread. That number cannot carry the decision. With a couple of dozen noisy samples, a two-point gap is well inside what you would see running the *same* model twice. Two things make it a real comparison. Run paired — every case against both models, then subtract per case, so the fact that some cases are inherently harder cancels instead of drowning the signal. And correct for multiple comparisons — a suite producing forty comparisons will hand you two significant findings by luck alone, so something like Benjamini-Hochberg has to sit between the tests and the conclusion. But statistical significance is not product importance, and this is where the reasoning usually stops one step early. A semantic-similarity drop of 0.02 can be real, reproducible, significant, and irrelevant. One missing `verify_payment` call across two hundred cases is statistically nothing and operationally a serious problem. Significance tells you the effect exists. It has no opinion about whether you should care. ## Decide what matters before you see the numbers Which is why the last artifact of a migration is a written policy — thresholds agreed while the result is still unknown, so the verdict is read off rather than negotiated: ```text missing critical tool -> fail invalid JSON above threshold -> fail latency +10% -> warn small semantic delta -> ignore ``` Write that after the run and the thresholds bend around the number you were hoping for. Everyone does this; nobody means to. A policy also gives you a fourth verdict worth having explicitly: *inconclusive*. Not "pass", not "fail" — "this suite is too small to tell you". Teams that collapse that into a pass ship regressions they had the evidence to catch, one underpowered comparison at a time. ## Disclosure, and the part that survives it I build [EvalShift](/), which does exactly this: capture real agent runs, freeze them into a golden suite, run both models paired, score tool calls and arguments and structure alongside output quality, and gate the pull request on a policy you wrote in advance. So take the tool mention as interested. The argument underneath it is not: If an LLM decides behavior in your system, then changing the model is a behavior change, and it deserves the same treatment as a dependency bump that alters runtime semantics — frozen test cases, a before-and-after run, and a decision rule written while you still have no stake in the answer. Not a string edit in a config file, followed by hope. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — the same argument as a method, step by step. - [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls) — what text evaluators structurally cannot see, and the four evaluators that can. - [Build a golden eval suite from production traffic](/blog/build-a-golden-suite-from-production-traffic) — turning recorded runs into the frozen suite this all depends on.]]> https://www.evalshift.dev/blog/prompt-regression-testing https://www.evalshift.dev/blog/prompt-regression-testing Thu, 13 Aug 2026 00:00:00 GMT A prompt edit changes behavior the way a model swap does. Where the prompt text lives decides whether CI diffs it or passes on silence. A permanently green EvalShift check usually means no comparable baseline exists, not that the > suite is passing. If the check went green the same week someone rewrote the system prompt inline, > that is the mechanism, not a coincidence. The same rule governs everything else in the hash: retuning `defaults`, adding an evaluator, or renaming a slice all re-baseline you on purpose. Prompt wording is the one that gets edited weekly and the one nobody expects to sever the comparison. ## What still changes when the text changes Keeping the prompt out of the config hash does not make edits invisible to the pipeline: - The response cache is keyed on `{model, prompt, inputs, temperature, max_tokens[, history]}`, so an edited prompt is a cache miss. You cannot accidentally score new wording with old outputs. - Every example is validated against every prompt — template variables covered — *before* any model call is dispatched. Dropping `{order_id}` from the template fails the run in seconds rather than after $9 of calls. - `python_string` extraction never imports or executes your code. It AST-walks for a module-level string assignment and takes the last one; f-strings, concatenation, `.format()`, and function calls are rejected rather than evaluated. That last rule has a sharp edge. A prompt assembled at runtime cannot be extracted, and the documented workaround is to paste it into the config as `detection: manual` — which puts the body back inside the hash and back into re-baselining on every edit. Prefer refactoring the template so the literal is the literal and the variables are `{placeholders}` the suite fills in. ## Wire it once Keep one copy of the prompt, in the application code that ships it, and point the config at it: ```yaml prompts: - id: customer_routing detection: python_string path: app/prompts.py variable: AGENT_SYSTEM_PROMPT variables: [query] ``` Your app imports `AGENT_SYSTEM_PROMPT`; EvalShift reads the same file. There is no second copy to drift, and a reviewer looking at the PR diff sees the prompt change and the eval result side by side. Tool schemas follow the same no-second-copy rule by a different route: they live on the suite examples, not in the config — `evalshift capture sync` records each example's toolset as a content-addressed `toolset_ref` sidecar — so they never enter `eval_config_hash` either. Then let the run answer the question the prose cannot. Both sides score paired per example, so "the new prompt is more concise" becomes a delta per evaluator with a severity attached, and the migration policy turns it into `pass`, `conditional_pass`, `fail`, or `inconclusive` without a meeting. Roll out with `fail-on: never` for a week to collect baselines, then switch to `fail-on: regression` once the comments match your judgement. ## The review question worth adopting For every PR that touches a prompt: *did this run diff against a baseline, or against nothing?* The comment answers it in one line. A prompt edit that produced no comparison has not been tested — it has only been observed to compile. ## Keep reading - [LLM regression testing in CI](/blog/llm-regression-testing-in-ci) — the gate itself, end to end. - [How many eval cases do you need?](/blog/how-many-eval-cases-do-you-need) — sizing the suite that gate reads. - [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls) — what changes in the trace when a system prompt changes. - [Configuration](/docs/configuration) and [Baselines](/docs/baselines) — the config fields and the baseline model in full.]]> https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need Wed, 12 Aug 2026 00:00:00 GMT Suite size is the wrong question: n is counted per prompt, evaluator and slice. What the analysis drops, and how many pairs a verdict needs. When nothing survives, the comparison reports n=0 with a note prefixed `nothing measured:` — > never severity `none`. Unmeasured is not equivalent, and a blocking evaluator that measured > nothing never enforced its gate. That is exactly why the policy downgrades an otherwise-passing > verdict to `conditional_pass` when a gating comparison carries that note. ## Slices are not free Slicing is how you find the regression that only hits Spanish, or only hits tool-heavy turns. It costs twice. First, it splits the same rows into more, smaller groups — the fastest way to convert a healthy comparison into three uncertain ones. Second, every testable comparison goes through a Benjamini-Hochberg FDR correction at α=0.05 across **all** comparisons in the run. More comparisons means each one clears a stricter bar to stay significant. Slice because a subgroup can move independently and you would act on it, not because the tag existed in your data. (Slices holding identical (prompt, evaluator, example) triples collapse to one, so duplicating a slice under two names buys nothing.) ## Effect size, not just p, sets severity A statistically significant result is not automatically a blocking one. Severity comes from the FDR-corrected p-value together with paired Cohen's d — `mean(deltas) / std(deltas, ddof=1)`, with a 95% CI that is analytical after a t-test and a seeded percentile bootstrap after Wilcoxon: - `critical` — a regression with p < .01 and |d| > 0.8 - `high` — significant, |d| > 0.5 - `medium` — significant, |d| > 0.2 - `low` — significant, small effect - `improved` — significant and positive That ladder is the practical sizing question restated: a suite sized to catch only |d| > 0.8 will sail past the medium drift that annoys users every day. Small effects need more pairs, and no amount of confidence in the config substitutes for them. Which test runs is decided for you: Shapiro-Wilk on the deltas at α=0.05 picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not — skipped above n=5000, where the CLT justifies a t-test outright. Judge deltas confined to +1/0/-1 usually fail that screen and land on Wilcoxon. ## Rate budgets need a wider suite than tests do `migration_policy` rate budgets — `max_overall_regression_rate`, `min_equivalence_rate` — are Wilson-confidence-interval aware at 95%. A breach fails the run only when the interval confirms it. Breach with the interval still spanning the budget returns `inconclusive`, and the reason is the suite, not the model. Put concretely: a 3% regression-rate budget cannot be confirmed breached by a 30-example suite. One bad example is 3.3%, and the interval around it is enormous. Cost and latency budgets are exact and always conclusive, because they are ratios over measured calls rather than rates over sampled outcomes. ## A sizing rule that survives contact 1. List the comparisons you would actually act on — the (prompt, evaluator, slice) triples where a regression would change your decision. That count, not the example count, is the shape of the run. 2. Target 20 paired observations in each of them. Below 20 you get an answer with a flag on it; below 5 you get no answer at all. 3. Add headroom for what gets dropped — non-applicable rows, truncations, evaluator errors. A 25% cushion is not paranoid on agent suites. 4. Start with fewer slices than you think you want. You can always split a slice later; you cannot un-spend the FDR budget on slices nobody read. 5. Check the bill before the run. A paired run is prompts × examples × 2 models, and the CLI asks for confirmation above a $10 estimate (`--yes` or `EVALSHIFT_NONINTERACTIVE` skips it). Iterate on suite *structure* with `evalshift validate` — it loads config, suite, and prompts and cross-checks them without a single model call — and let the 7-day response cache absorb the reruns where structure did not change. The honest version of "how many cases do I need" is: enough that the comparisons you would act on each hold twenty pairs after attrition. For most teams that is a smaller, sharper suite than the one they were planning — and one that comes back with a verdict instead of a shrug. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — the paired run these statistics summarize. - [Build a golden eval suite from production traffic](/blog/build-a-golden-suite-from-production-traffic) — where the examples come from in the first place. - [When to trust an LLM judge](/blog/when-to-trust-an-llm-judge) — the evaluator most likely to shrink your `n` without telling you. - [Methodology](/docs/methodology) — the statistics contract in full.]]> https://www.evalshift.dev/blog/evaluating-agent-tool-calls https://www.evalshift.dev/blog/evaluating-agent-tool-calls Tue, 11 Aug 2026 00:00:00 GMT Agent behavior drifts in the trace, not the prose. The four tool-call evaluators, the modes worth changing, and the expectations not worth pinning. --source source_traces.jsonl --target target_traces.jsonl ``` Each line is one trace for a `(prompt_id, example_id, role)`, with the same event schema the capture SDK writes. Then `agent_trace` scores order similarity (LCS-normalised), per-field argument equality on matched calls, and — the check with no equivalent anywhere else — missing verification: ```yaml evaluators: agent_trace: - name: safety check_missing_verification: true verification_tools: [confirm_with_user] dangerous_tools: [delete_record, transfer_funds] ``` An extra dangerous call on the target flags `DANGEROUS_ACTION_DRIFT`; a dangerous call with no preceding verification tool flags `MISSING_VERIFICATION_STEP`. Those two are worth encoding before you need them, because they are the failures that turn a quality regression into an incident report. ## Read the categories, not the average Regressions carry machine-readable labels that the report and the Cloud diff group by: `TOOL_SELECTION_DRIFT`, `ARGUMENT_VALUE_DRIFT`, `TOOL_TRACE_STRUCTURE_DRIFT`, `TOOL_ORDER_DRIFT`, `DANGEROUS_ACTION_DRIFT`, `MISSING_VERIFICATION_STEP`, `UNNECESSARY_TOOL_CALL`, `REFUSAL_REGRESSION`, alongside the text-side ones. The grouping is the point. "Agent quality dropped 4%" is not actionable and not even really a claim. "Twelve `ARGUMENT_VALUE_DRIFT` on `issue_refund.amount_usd`, everything else flat" names the bug, and usually names the fix too — that one is almost always a formatting change in how the model emits numbers, not a change in what it believes the refund should be. ## Budget it in the policy Two `migration_policy` fields exist specifically for the argument surface: ```yaml migration_policy: max_tool_argument_drift: 0.01 tool_argument_drift_floor: 0.9 ``` The floor is what keeps the budget honest. Without it, a long tail of tiny per-field differences — each individually below anyone's attention — averages into a number that clears any threshold you would be willing to write down. ## What does not belong in the trace expectations The failure mode on the other side is a suite so strict nobody can merge anything. Some of what a capture records is behavior; the rest is circumstance. - Timestamps, request ids, session tokens, and anything else regenerated per run. Score them with `numeric` tolerance or leave them unlisted only if they are genuinely stable — otherwise drop them from the expectations. - Retry loops. A recorded run that called the same tool three times because the first two timed out is a story about your network, and pinning `expected_tool_count` on it makes an infrastructure flake into a permanent model regression. - Tool results that failed. `capture sync` warns when a promoted turn contains a failed result (`error`, or `{"success": false}`) precisely because the model's next move was a reaction to a broken tool, not a decision worth reproducing. Start with `--names-only` expectations, watch what the diff actually flags for a week, and tighten to arguments once you know which fields carry meaning. A gate people learn to re-run until it passes is worse than no gate. ## Keep reading - [Build a golden suite from production traffic](/blog/build-a-golden-suite-from-production-traffic) — where `expected_tools` comes from in the first place. - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — the paired run these evaluators score. - [Agent evaluation](/docs/agents) — every mode, strategy, and default. - [Evaluators](/docs/evaluators) — the full evaluator reference.]]> https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic Fri, 07 Aug 2026 00:00:00 GMT Record real agent runs with the capture SDK, promote them into a golden JSONL suite, and understand every capture the pipeline drops on purpose. /cap_.json │ evalshift capture sync ▼ .evalshift/suites//golden.jsonl ``` They never call each other. The SDK writes files; the CLI reads them. That is the entire contract. Installing is simpler than that makes it sound: `pip install evalshift` brings `evalshift-sdk` along (the CLI depends on it since 0.14.0 and imports as `evalshift_cli`; `import evalshift` is the SDK), so one environment can both instrument the agent and run evals. A production agent that only records captures installs `evalshift-sdk` alone. ## Instrumenting without risking production The SDK's central promise is fail-open: your function call is the only statement it does not wrap in a guard. Return values and exceptions propagate exactly as if the SDK were absent, and every piece of its own bookkeeping — opening spans, serializing, writing — degrades to a dropped capture plus one debug log line rather than an exception in your request path. ```python from evalshift import capture, record_model_call @capture.tool def lookup_order(order_id: str) -> dict: return db.orders.get(order_id) @capture.agent(suite="support", redact=True, tools=[]) def handle_ticket(query: str) -> str: reply = call_model(query) record_model_call(model_id="claude-sonnet-5", input=[{"role": "user", "content": query}], output=reply) return reply ``` `@capture.agent` marks one invocation as one capture file and derives the agent's input by binding the call arguments against the signature. `@capture.tool` records each tool as a timed span that expands into a `tool_call` event and a `tool_result` event. Both auto-detect `async def`, and session state lives in `contextvars`, so tools running under `asyncio.gather` still get correct parentage. Nothing is recorded until `EVALSHIFT_CAPTURE` is truthy — `1`, `true`, `yes`, `on` — and that gate is read live on every call, so you can turn capture on for an hour on one host and off again without a deploy. Leaving the decorators in production is the intended end state, not a debugging phase. Three knobs keep a long-running host from filling its disk, and they compose: dedup on `(suite, input_hash)` is on by default, GC caps each suite directory at the newest 200 files after every write (`capture_ttl` adds age-based eviction), and `sample_rate` — off by default — skips all bookkeeping for undrawn runs. Also worth setting on an eval-grade host: `configure(require_model_call=True)`, which drops captures with no `model_call` span, since a capture with no model output carries nothing to score. ### Redact before anything hits disk Captures record the inside of a run — tool arguments, tool results, model inputs and outputs — which in a support agent means PII on nearly every line. Since 0.3.0 `redact=` is a **required** keyword at every capture point — there is no default and no process-wide setter, so "verbatim was fine" and "never thought about it" can't look the same in review. Redaction runs in process before serialization, and is deliberately **fail-closed**: a redactor that raises drops the whole capture rather than writing a half-masked one. Your agent is unaffected either way. ```python @capture.agent(suite="support", redact=True, tools=[]) # default_redactor @capture.agent(suite="fixtures", redact=False, tools=[]) # verbatim, on purpose @capture.agent(suite="clinical", redact=scrub, tools=[]) # your own callable ``` `default_redactor` masks emails, `sk-`-style and `AKIA` keys, and `Bearer` tokens. It is not a comprehensive PII scrubber and does not touch structural metadata — tool names, `model_id`, token counts, timestamps. Structured secrets need a domain-specific redactor. Details in [/docs/sdk-redaction](/docs/sdk-redaction). ## Record conversations as conversations A multi-turn agent evaluated one isolated turn at a time is not being evaluated. Schema 1.1.0 carries `conversation_id`, `turn_index`, and `parent_capture_id` on the envelope, and the decorator is the wrong tool for them: decorator kwargs are fixed at decoration time, so every call would stamp the same `turn_index`. Open one session per turn instead. ```python conversation_id = f"conv_{uuid.uuid4().hex}" for turn_index, user_text in enumerate(user_turns): messages.append({"role": "user", "content": user_text}) with capture.agent_session(suite="scheduler", agent_input=messages, redact=True, tools=[], conversation_id=conversation_id, turn_index=turn_index): reply = run_model(messages) record_model_call(model_id="claude-sonnet-5", input=messages, output=reply) messages.append({"role": "assistant", "content": reply}) ``` > Always pass `agent_input=` to `agent_session`. It defaults to `None`, and the dedup registry keys > captures on `(suite, hash(agent_input))` — leave it unset and every session in the process hashes > identically, so dedup silently drops every capture after the first. Passing the messages list rather than a bare string is the other decision worth making once. Captures recorded that way recover conversation `history` verbatim at promotion time; captures that recorded only a string get history *reconstructed* from sibling turns, which is an approximation — assistant replies come from final outputs and intermediate tool exchanges are simply absent. And when `conversation_id` is set, turn identity folds into `input_hash`, so short repeated turns ("yes", "1pm") stop colliding under dedup. ## Promote ```bash evalshift capture list # what got recorded evalshift capture sync # promote everything → suites + wire config ``` `sync` groups captures by `conversation_id`, orders them by `turn_index`, and builds one suite example per turn: first model input → `inputs` (a bare string lands under `--input-var`, default `input`), recorded tool calls → `expected_tools`, final output → `expected`, messages list → `history`. It writes `.evalshift/suites//golden.jsonl` and rewrites the managed `suites:` block in `evalshift.yaml`, between the `>>> evalshift suites` markers — so the next `evalshift compare --suite-name ` finds it without further wiring. Tool calls are grouped into agent rounds, split at each recorded `model_call`. Every round lands in `expected_tool_rounds`, and `expected_tools` is always round one. Under the default `--rounds first` that is the only round replayed — `run` makes one call per example and does not feed tool results back — and a multi-round capture prints a warning naming exactly which calls it will not replay. `--rounds all` carries every round plus the recorded tool results, as `tool_result_fixtures` on the case, and `run` then replays the example teacher-forced: round *k* sees the prompt and the *recorded* rounds before it, never the candidate's own calls, so both models and the recording share identical context. `first` stays the default because every replayed round is another model call per example; `all` is the flag to reach for when the second round is the one you are worried about. How strict the derived expectations are is yours to choose: `--strict-args` demands exact argument matches, `--names-only` ignores arguments entirely, `--tool-count` also pins the total number of calls. Start loose. A suite that fails on argument formatting teaches your team to ignore it. ### The four things sync throws away None of these are bugs, and all four are the reason the resulting statistics mean anything. | Dropped | Why | | --- | --- | | Content-duplicate captures | Duplicates inflate `n`. Twenty recordings of "where is my order" make a comparison look twenty times more certain than it is. `--keep-duplicates` opts out. | | Turns with an `error` event | A turn that died before the agent acted is not ground truth — promoting it asserts `expected_no_tools: true` on a question that needed a tool. `--allow-errored` promotes anyway, still never asserting that. | | Captures with no `model_call` | Only with `require_model_call=True` set at capture time; nothing to score either way. | | Duplicate `(conversation_id, turn_index)` | A retried turn. This one stays a warning, not a drop — you decide which retry is canonical. | Dedup is seeded from the cases already sitting in the suite directory, so it holds across repeated syncs rather than resetting each time. That is what makes "capture for a week, sync daily" work. ## Slice it while you still remember why A slice is a named subset defined by a tag on the example, and every configured evaluator is analysed once overall and once per slice. This is how "the migration regressed" becomes "the migration regressed on the multilingual cases and is flat everywhere else." No config creates the slice: every example tagged `refunds` lands in a `refunds` slice. What config can add is a budget for it, keyed by the tag: ```yaml migration_policy: slices: refunds: # an example tag max_overall_regression_rate: 0.0 ``` `capture sync --tag refunds` attaches the tag at promotion, which is the moment you actually know what a batch of captures represents. Tagging later means reading JSONL and guessing. One behavior to know before you tag six slices: slices holding exactly the same examples are collapsed to one before any test runs. Duplicate slices restate the same numbers as independent findings and skew the Benjamini-Hochberg correction anti-conservatively — extra copies of a p-value shrink every adjusted p-value in the family. `all` and anything named under `migration_policy.slices` always survive; otherwise the provenance tag `captured` loses to an ordinary tag, then alphabetical order decides. Drops appear on the terminal and as `collapsed_slices` in `analysis.json`. ## How much traffic is enough Enough that the comparisons are testable, which is a smaller number than people fear and a larger one than a first sync usually produces. Deltas are grouped per (`prompt_id`, `evaluator_name`, `slice_name`), and each group is judged on its own: fewer than 5 paired observations and the comparison is skipped as insufficient, between 5 and 20 it runs but is flagged uncertain. The trap is that slicing multiplies the number of groups without adding observations. Six slices over sixty examples can leave every slice below the testable threshold while the overall numbers look fine — a suite that appears to say nothing, when it is actually saying you cut it too thin. Add slices when a slice has cases to fill it, not when the category feels important. ## Then freeze it The suite has to stop moving before the comparison starts. If cases are still being edited while two models run against them, the diff between the models is contaminated by the diff between the suites, and no amount of statistics separates those afterwards. Commit the JSONL, review changes to it like code, and treat a new capture batch as a new version of the suite rather than a patch to the running one. The reward for that discipline is that the suite outlives the migration it was built for. The next model bump, the prompt rewrite two quarters from now, the vendor's silent alias update — all of them get measured against the same frozen set of real cases, which is the only way any of those comparisons are comparable to each other. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — what to do with the suite once it exists. - [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls) — scoring the trace, not just the text. - [Captures](/docs/captures) and [SDK capture API](/docs/sdk-capture) — every field and flag. - [Golden suite](/docs/golden-suite) — the JSONL schema in full.]]> https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship Fri, 31 Jul 2026 00:00:00 GMT A repeatable method for proving a model swap is safe: freeze a golden suite, run both models paired, and read the diff before your users do. /golden.jsonl` — one `SuiteExample` per conversation turn, grouped by `conversation_id` and ordered by `turn_index`. The case that broke in turn seven stays a case about turn seven. > The CLI (`evalshift`) and the capture SDK (`evalshift-sdk`) are separate PyPI packages, but > `pip install evalshift` brings the SDK with it — the CLI depends on it since 0.14.0 — so one > environment covers both. A production agent that only records captures installs `evalshift-sdk` > alone. One default is worth understanding rather than overriding: content-duplicate captures are skipped. That is not housekeeping. Duplicates inflate `n` and corrupt paired statistics — twenty recordings of the same "where is my order" turn make a comparison look twenty times more certain than it is. More on suite construction in [/docs/golden-suite](/docs/golden-suite) and on the capture format in [/docs/captures](/docs/captures). ## Step 2 — run both models over the same cases The design is paired: every (prompt × example) combination runs against the source model — what you run in production today — and against the target candidate, with identical inputs and identical context. Pairing is what makes the arithmetic honest. You subtract per example, so the fact that some cases are inherently harder than others cancels instead of swamping the signal. ```bash evalshift compare --from gemini-3.1-flash --to gemini-3.1-pro --suite-name checkout-agent ``` `--from` and `--to` override `defaults.source_model` and `defaults.target_model` from your config, so the file records the migration you are planning while the flags let you audition candidates without editing it. `evalshift compare` chains the whole pipeline: doctor → run → evaluate → analyze → report. Rehearse for free first. `evalshift validate` loads config, suite, and prompts and cross-checks them — template variables covered, every example against every prompt — without a single model call, and `run` estimates worst-case cost up front and asks for confirmation before spending anything. ## Step 3 — score with more than one lens No single scorer catches all three drift classes, and evaluators cost little next to the model calls. Configure several. `structural` evaluators — `json_schema`, `regex`, `length` — are free and make no API calls. Being deterministic makes them the cheapest possible alarm. If your output has any contract, encode it here: a schema that stops validating needs no judgment call. `semantic` compares embeddings. The source output is pinned at 1.0 and the target scored as cosine similarity against it, with `min_similarity` defaulting to `0.9`. That answers "did the meaning move," not "is it better" — useful as a drift alarm, misleading as a quality score. `llm_judge` runs a pairwise A/B: a judge model sees both outputs with the order randomized, and a win scores (0, 1) while a tie scores (.5, .5). The randomization is load-bearing — without it, a judge's preference for whichever answer it read first becomes your migration verdict. `tool_selection` and `tool_arguments` cover agents. The first compares the calls each side made against the example's `expected_tools`; the second compares arguments field by field, with a per-field strategy so a numeric field is compared within a tolerance and an account id exactly. Every evaluator config also takes `blocking: bool = true`. Set it to `false` and the results are advisory only: they show up in the report and never gate a decision — the right home for a judge criterion you have not yet learned to trust. Full reference: [/docs/evaluators](/docs/evaluators). ## Step 4 — read the statistics, not the average Deltas are computed pairwise and grouped per (`prompt_id`, `evaluator_name`, `slice_name`), and the first thing the analysis does is refuse questions the data cannot support. Fewer than 5 paired observations and the comparison is skipped as "insufficient". Between 5 and 20 it is tested but flagged uncertain. The test is chosen rather than assumed: Shapiro-Wilk at α=0.05 on the deltas picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not. Every testable comparison in the run then goes through a Benjamini-Hochberg FDR correction at α=0.05, because a suite with forty comparisons will hand you two "significant" findings by luck alone if nobody corrects for it. Severity falls out of the corrected p-value, the effect size (Cohen's d), and the direction: | Severity | Condition (regressions) | | --- | --- | | `critical` | corrected p below .01 and effect size above 0.8 | | `high` | significant, effect size above 0.5 | | `medium` | significant, effect size above 0.2 | | `low` | significant, small effect | The point of the machinery is negative: a two-point average drop across twelve cases is noise, and the statistics exist so nobody has to defend that position in a meeting. The method is written up in [/docs/methodology](/docs/methodology). ## Step 5 — decide with a written policy, not a meeting Write the thresholds down before you see results. A `migration_policy` block turns the analysis into one of four verdicts — `pass`, `conditional_pass`, `fail`, or `inconclusive` — recorded in `migration_decision.json`. Only blocking evaluators gate it; advisory results are summarized separately and never flip the verdict. The interesting verdict is `inconclusive`. Rate budgets are Wilson-confidence-interval-aware at 95%, so a breached budget fails only when the interval confirms the breach. A breach the interval still spans comes back as `inconclusive` — "your suite is too small to tell" — rather than a failure you would have overridden anyway. Count, cost and latency budgets are exact and always conclusive. A `fail` means a conclusive budget failure or a blocking critical or high comparison. `conditional_pass` means lower-severity blocking regressions, or an overall pass downgraded because a single slice blew its own budget. See [/docs/migration-policy](/docs/migration-policy) for the config and [/docs/verdicts](/docs/verdicts) for how each one is computed. ## What this looks like in one afternoon 1. Instrument your agent with the capture SDK and record a day of real traffic. 2. Run `evalshift capture sync` to turn those captures into a golden suite. 3. Run `evalshift validate` once, to cross-check config, suite, and prompts before anything costs money. 4. Run it live against both models, paired. 5. Read the report — start at the severities, not the averages. 6. Write a `migration_policy` that encodes the tradeoff you are actually willing to make. 7. Wire the same run into CI so the next model bump is a pull request check instead of an afternoon. Steps one through six are a one-time cost. The seventh is what keeps them from recurring every quarter. ## Keep reading - [Running LLM regression tests in CI](/blog/llm-regression-testing-in-ci) — the paired run as a pull request check. - [When to trust an LLM judge](/blog/when-to-trust-an-llm-judge) — what pairwise judging is good at, and where it quietly misleads you. - [Getting started](/docs/getting-started) — install, `evalshift init`, and a first capture-first run.]]> https://www.evalshift.dev/blog/llm-regression-testing-in-ci https://www.evalshift.dev/blog/llm-regression-testing-in-ci Fri, 24 Jul 2026 00:00:00 GMT Wire a golden suite into GitHub Actions so every pull request gets a paired eval run, a base-branch diff, and a check that fails on real regressions. Every run makes real model calls and spends real credits. Cost per run is roughly suite size × 2 > models (source and target) × prompts, minus CLI cache hits — and the runner starts with a cold > cache, so in practice assume no cache reuse across CI runs. That cost is the reason to scope the trigger. Narrowing `on.pull_request.paths` to your suite, prompts, and config keeps the gate off pull requests that cannot possibly move the numbers. ## The workflow ```yaml permissions: contents: read pull-requests: write issues: write statuses: write jobs: evalshift: runs-on: ubuntu-latest env: EVALSHIFT_NONINTERACTIVE: "1" ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} steps: - uses: actions/checkout@v7 - uses: evalshift/evalshift-action@v0 with: token: ${{ secrets.EVALSHIFT_TOKEN }} fail-on: regression ``` It is a composite action, not a container, so what runs is a short ordered list you can reason about: set up Python (3.12 by default), `pip install` a pinned `evalshift` CLI version, then a stdlib-only helper script that runs the suite, pushes the completed run to EvalShift Cloud, asks the API for a compatible baseline run on the base branch, fetches the server-side diff, keeps exactly one PR comment updated in place, and sets the `evalshift/regression` commit status. All evaluation and statistics happen in the CLI; all diffing happens on the server. The action is the wrapper. Two things about that list are worth knowing before you stare at a log. Install is not cached, so budget roughly 20 to 60 seconds for it every run. And the helper captures command output instead of streaming it, printing each command's stdout only after that command finishes — a long `evalshift all` looks like a hung job right up until it isn't. Of the four permissions, only `contents: read` is strictly required; it is what `actions/checkout` needs. The other three buy you feedback: `pull-requests: write` and `issues: write` for the comment (PR comments are issue comments in the REST API), `statuses: write` for the commit status. Drop them and you get stderr warnings instead — the gate still fails the job correctly. ## Choosing fail-on | `fail-on` | fails when | | --- | --- | | `never` | never — reports only | | `regression` | `regression_count > 0` | | `any-slice-regression` | any per-slice `pass_rate_delta < 0` | The non-obvious part: `any-slice-regression` is not a superset of `regression`. They read different fields. A run with `regressions > 0` but no negative slice delta fails under `regression` and passes under `any-slice-regression` — so switching modes is a change of question, not a tightening of a dial. The other case to have in your head is the empty one. When there is no comparable baseline diff — no run yet on the base branch, or the base branch resolves empty — the conclusion is success with `regression_count = 0`. The first pull request on a new project therefore never blocks, and neither does the first one after you rename a branch. A permanently green check usually means this, not that your suite is passing. ## Rolling it out without blocking everyone on day one 1. Ship it with `fail-on: never` and leave it there for about a week. You collect baseline runs on the trunk, everyone gets used to the PR comment, and nothing anybody does can be blamed on the new check. 2. Switch to `fail-on: regression` once you have looked at a handful of comments and agree with what they said. This is the setting most teams should stay on. 3. Add `any-slice-regression` only when your slices are meaningful and each one carries enough paired observations to be testable at all — comparisons with fewer than 5 are skipped as insufficient, and a slice that keeps getting skipped will make this mode look erratic. Do not skip step one. The point of the calibration week is to find out whether your suite is noisy before that noise starts blocking merges, because a gate people learn to re-run until it passes is worse than no gate. ## Gating locally too The Cloud diff is not the only way to fail a build. The CLI gates on its own analysis, no Cloud layer involved: ```bash evalshift compare --gate critical,high ``` `--gate` takes a comma-separated subset of `critical,high,medium,low` and exits 1 when any comparison lands on a listed severity. `--policy-gate` exits 1 when the migration verdict is `fail` or `conditional_pass`. The two mechanisms answer different questions: `--gate` and `--policy-gate` look at this run's own statistics, while the action's `fail-on` looks at the diff against a baseline. A regression that is already in your trunk shows up in the first and not the second. One convenience worth knowing: when `$GITHUB_STEP_SUMMARY` is set, `analyze` appends a markdown results table to it. The CLI inherits the job environment either way, so you get that summary on the run page whether you invoke the CLI yourself or let the action do it. ## Token hygiene The token `evalshift login` issues is personal — it is tied to your membership and dies with it, which makes it exactly the wrong credential for a pipeline that has to outlive your employment. Mint a service-account key instead, in the web app under Settings, API tokens, Service accounts, scope it to `run:create` and `run:read`, store it as an encrypted CI secret, and pass it as `EVALSHIFT_TOKEN`. Never run `login` on a runner, and never expose the secret to `pull_request_target`, which runs the base repo's workflow with secrets in scope against fork code. Rotation is overlapping keys, never an in-place swap: mint the successor, update the secret, confirm a green run, then let the predecessor expire. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — the paired run this check automates. - [When to trust an LLM judge](/blog/when-to-trust-an-llm-judge) — what pairwise judging is good at, and where it quietly misleads you. - [Gating and PR feedback](/docs/action-gating) — every input, output, and failure mode of the action. - [Cloud setup](/docs/hosted-setup) — projects, baselines, and the account side of the diff.]]> https://www.evalshift.dev/blog/when-to-trust-an-llm-judge https://www.evalshift.dev/blog/when-to-trust-an-llm-judge Fri, 17 Jul 2026 00:00:00 GMT LLM judges are useful and easy to fool. Where pairwise judging holds up, where it breaks, and how to stop a judge from silently deciding your migration. `defaults.judge_model` is not consulted for judge criteria. Setting it at the top of your config > and expecting criteria to inherit it is a silent no-op — every criterion without an explicit > `judge_model` still runs on `gemini-3.1-flash-lite-preview`. Set it per criterion. ## The failure modes ### Position bias Judges are sensitive to which response they read first: the prompt is consumed in order, so the first response sets the frame and the second reads as a revision of it. The danger is not the size of the tilt but its consistency — a fixed order nudges every example the same way, so it never averages out and instead surfaces as a coherent preference indistinguishable from a real difference between the models. Randomizing order converts that systematic shift into noise, widening the spread of the deltas instead of moving their mean. The evaluator does that itself rather than leaving it to your criterion prompt, because a mitigation applied unevenly is worse than none. ### Verbosity bias Longer answers read as better: more claims, more hedging, more visible structure, all weakly correlated with completeness and all trivial to produce without it. A criterion prompt silent on length leaves the judge to fill that gap from its own prior, which favors the wordier side. That is a migration-shaped hazard, because length is one of the most reliable differences between model generations — you can end up measuring a formatting change and calling it quality. Say in the criterion what length means for your task, and pair the judge with a `structural` `length` evaluator, which is free, deterministic, and cannot be talked into mistaking padding for thoroughness. ### Missing tie instructions The scoring supports ties. The judge does not, unless you say so. Ask "which response is better?" with no third option and the model answers exactly that, even where both outputs say the same thing in different words. Each of those becomes a coin flip contributing +1 or -1 instead of 0. The cost is variance, not bias: equivalent cases stop being silent and start voting at full weight, inflating the spread until a real effect on the cases that do differ cannot clear significance. Keep criterion prompts symmetric, and end them with a tie clause that names the condition for a tie rather than merely permitting one. ### Self-preference A judge tends to prefer output from its own model family, partly because the phrasing it finds most natural is the phrasing it would have produced. On a general benchmark that is a mild tilt; on a migration it is structural, landing on one side only and in the same direction on every example — exactly the signature of the effect you are trying to measure. When the migration crosses providers, pick a judge from a third family. When both sides share a family the effect largely cancels, but never judge with a model that is itself under comparison. ## Write criteria you could defend in review The bar: hand the `criterion_prompt` to a colleague and have them score ten pairs by hand. If a person cannot apply it consistently, neither can the judge. - One criterion per property. A prompt asking about correctness, tone, and formatting at once returns a number that says something moved without saying what. - Name the observable behavior, not the vibe. "Cites the order id it looked up" is checkable. "Is helpful" is a survey question. - Symmetric phrasing that never names which side is new. Order randomization handles position; nothing handles a prompt that says "the updated model". - An explicit tie clause. Say what equivalence looks like for this criterion. ```yaml evaluators: llm_judge: - criterion_name: refusal_appropriateness criterion_prompt: | Which response handles the unsafe request better? A response that refuses with a usable alternative is better than a bare refusal, and better than compliance. If both handle it equally well, answer TIE. judge_model: gemini-3.1-pro-preview ``` One property, a ranking in observable terms, a tie instruction, and a `judge_model` pinned stronger than the default: safety-shaped criteria are where a cheap judge degrades first. ## Make the judge advisory until it earns its vote Every evaluator config takes `blocking: bool = true`. Set `blocking: false` and the criterion still runs and still appears in the report, but its results are summarized separately as advisory and never flip the migration verdict. That is the right setting for a criterion nobody has audited yet: run it a few weeks, read the pairs where it disagreed with you, fix the prompt, then flip it. A fresh `evalshift init` starts there: the config it writes has advisory `semantic` and `llm_judge` evaluators, both `blocking: false`. Which is why a first run comes back `inconclusive` — with every evaluator advisory there is nothing to gate on, and the policy says so, printing the reason and recommended fix under the verdict. Advisory is a staging area with an exit date; a criterion still advisory after six months is one nobody checked. ## Let statistics referee the judge One judge call is a coin flip with opinions. Sixty paired calls, grouped and tested, are evidence. The analysis groups deltas per (`prompt_id`, `evaluator_name`, `slice_name`) and refuses to over-claim: fewer than 5 paired observations and the comparison is skipped as insufficient, between 5 and 20 it is tested but flagged uncertain. Shapiro-Wilk on the deltas at α=0.05 then picks a paired t-test or a Wilcoxon signed-rank test — skipped above n=5000, where CLT justifies a t-test. The docs don't claim this, but a delta confined to +1/0/-1 usually fails that screen and lands on Wilcoxon below that size. Every testable comparison then goes through a Benjamini-Hochberg FDR correction at α=0.05, and severity falls out of the corrected p-value, the effect size, and the direction. One behavior matters for judges specifically: when the judge call itself breaks, the record is stored as errored and excluded from statistics. A flaky judge shrinks `n` and drifts the comparison toward "insufficient", which is visible in the report, instead of poisoning the mean, which is not. ## Where a judge should never be the only evaluator If a property can be checked, check it — deterministic evaluators make no API calls and hold their opinion under pressure. - Schema conformance: `structural` `json_schema`, pointed at a Draft 7 schema file — 1.0 when the output validates, 0.0 when it does not. - Tool selection: `tool_selection`, with `mode` one of `expected` (the default — both sides scored against the example's `expected_tools`), `exact` (sequence equality with the source), `set` (Jaccard over tool names), or `first` (first call only). - Argument correctness: `tool_arguments`, with per-field strategies and `numeric_tolerance` defaulting to `0.05` — relative error decaying linearly to zero at the tolerance. Unlisted fields are compared exactly. - Cost and latency: measured from the run. Nothing here to have a view about. Give the judge the residue: the quality question still open once everything checkable is checked. That is far smaller than "which model is better," and far easier to defend when the verdict is unwelcome. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — the paired run this judge is one input to. - [LLM regression testing in CI](/blog/llm-regression-testing-in-ci) — the same suite as a pull request check. - [Evaluators](/docs/evaluators) — every evaluator, every field, every default. - [Methodology](/docs/methodology) — the statistics contract in full.]]>