The evalshift CLI is a five-stage pipeline. You point it at a golden suite of examples; it runs the same prompts on two models — your current production model and the candidate — scores both sides with structural, semantic, LLM-as-judge, and tool-call evaluators, then runs paired statistics over the deltas and tells you, with confidence intervals and multiple-comparison correction, what actually regressed.

It is local-first: every stage runs on your machine and writes its artefact under .evalshift/runs/<run-id>/. The only network calls are the model API calls you asked for — and, if you opt in, pushes to the EvalShift Cloud.

The CLI is open source under Apache-2.0 — the code lives at evalshift/evalshift-cli ↗.

pip install evalshift installs the CLI (import package evalshift_cli; python -m evalshift_cli is the module form of the evalshift binary) and, since 0.14.0, the capture SDK it depends on (package evalshift-sdk, import name evalshift) — both live in one environment. Scripts that imported CLI internals from evalshift.* must import from evalshift_cli.

## The pipeline

init        →  doctor  →  run        →  evaluate      →  analyze          →  report
(scaffold)     (checks)   raw.jsonl     scores.jsonl     analysis.json        report.html
                          state.json                     migration_decision   report.json
                                                         .json (if policy)

• +doctor — validates config and shows which provider keys are visible. Missing keys are soft warnings. It also reports the toolset each configured suite carries, and flags a suite whose examples carry more than one distinct toolset — legal, but also the shape a wiring mistake takes. An evalshift-sdk row reports which package the evalshift import name resolves to (warn when the SDK is missing or shadowed by an older CLI’s leftover files; never a failure), a ci pin row warns when a workflow pins an older, newer, or no evalshift-version, and a judge family row warns when an llm_judge judge shares a provider with the source or target model (self-preference bias; advisory, validate prints the same line).
• +run — validates every example against every prompt, estimates cost, then dispatches (prompt × example × {source, target}) calls through an async orchestrator. Progress is checkpointed every 50 completions.
• +evaluate — scores each (source, target) pair with the configured evaluators.
• +analyze — paired statistics per (prompt, evaluator, slice), Benjamini–Hochberg FDR correction, severity classification, and the migration-policy verdict when configured. See Methodology.
• +report — renders a single-file HTML report (no external assets; works offline and attaches cleanly to a PR), topped by a plain-language narrative of the run: one summary each for the verdict, the advisory signal and the economics, plus behavioural findings taken from the worst regressions. The prose is machine-written; every figure in it is computed first, handed to the model pre-rendered, and validated on the way out, so the narrative cannot contain a derived or invented number. One extra model call, cached — skip it with --no-insights. The header shows an N samples per example pill when defaults.samples_per_example is above 1, and up to three banners above the verdict: models that no longer honour temperature (on a single-sample run the banner suggests samples_per_example), Constraints not honoured for generation parameters an arm could not accept, and a judge that shares a family with an arm. report.json carries samples_per_example, dropped_params and judge_family_overlap.
• +compare — chains everything end to end over one suite and adds --gate, --policy-gate, --push, --open. Formerly all; that name is hidden now but still works.

Every stage is independently re-runnable: each writes one artefact and the next stage reads it.

## Project setup: init

evalshift init is the entry point. It writes only a minimal, capture-first evalshift.yaml: a passthrough replay prompt, advisory semantic + LLM-judge evaluators, an empty managed suites: block for capture sync to fill, and a migration policy. The intended flow: instrument your agent with the capture SDK, record captures, promote them with evalshift capture sync, run against the promoted suite. Options: --provider, --profile (pre-tuned policy budgets: model-upgrade, cost-reduction, local-model, quantization, provider-switch), --ci (scaffold the GitHub Action), --wire-agents (write EVALSHIFT.md and point AI-agent files at it).

## Response cache

Live responses are cached in SQLite at ~/.evalshift/cache.db, keyed by a hash of (model, prompt, inputs, temperature, max_tokens, history), plus the generation_config, the tools array as sent, round_index and sample_index when set — a single-sample, single-round run keeps its existing keys — with a 7-day TTL. Re-running an identical evaluation makes no run-stage calls, agent suites included, and the evaluate stage’s embedding and judge caches apply too. Disable per-project with defaults.cache: false; wipe with evalshift cache clear.

An example with a non-empty toolset (toolset_ref or inline tools) gets one entry per replayed round, keyed on the canonical model, the prompt and inputs, the exact message list that round sends (history, current turn, and the recorded rounds and fixture results teacher forcing feeds back), the tool list exactly as sent, in order (tool names, descriptions, schemas and strict — reordering the tools is a miss), generation_config (so tool_choice and parallel_tool_calls), the effective temperature and max_tokens, the round index and the sample index. Editing a fixture re-sends only the rounds after it. A hit restores the parsed tool trace (call ids, arguments, final text, refusal), tokens, cost, latency and finish reason, so the raw.jsonl row matches the live one apart from cached, which is true only when every round was a hit, and cached_rounds. A row with any round served from the cache carries latency measured on an earlier run, so it stays out of the live latency figures and its latency delta is marked not comparable. Errors are never cached: the next run re-sends a failed round and serves the rounds before it from the cache. Truncated responses are cached and stay flagged. A cache.db written by an earlier version is upgraded in place and keeps its entries, and several evalshift processes may open it at once.

## Checkpointing and resume

state.json records a hash over the canonicalised config plus the suite path. run --resume picks up the most recent in-progress run, verifies the hash still matches (aborts if config or suite changed — a changed config would corrupt the pairing), and skips every call already present in raw.jsonl. Errored calls are counted as done in a resumed run — start a fresh run to retry them.

## Run retention

Run history is pruned automatically after every completed run, per suite: keep the newest retention.max_runs_per_suite (default 20), optionally evict runs older than retention.run_ttl_days. In-progress runs and the run just finished are never pruned. evalshift runs clean prunes on demand.

## Which models can I use?

Anything LiteLLM supports. A small built-in registry maps aliases to canonical provider/model ids, and unknown ids pass through with the provider inferred from the prefix (gemini-* → Google, claude-* → Anthropic, gpt-*/o1-*/o3-* → OpenAI, deepseek-* → DeepSeek). The registry never gates. Verify a model with evalshift test-call -m <id>.