This page is the data contract for EvalShift Cloud — what a push sends, field by field, and what never leaves your machine. If you need to clear EvalShift with a security or compliance team, this is the page to hand them. The CLI contains no telemetry: no analytics, no crash reporting, no phone-home of any kind.
The CLI opens exactly two kinds of network connections, both initiated by you:
- +Your model providers (whichever you configure — any provider LiteLLM supports), using your own API keys:
runsends the rendered prompts and conversation histories to both models,evaluatesends outputs to the embedding andllm_judgemodels, andreportsends the worst regressions' inputs and outputs todefaults.insights_modelunless you pass--no-insights. This traffic goes to your providers, never to EvalShift. - +EvalShift Cloud (
api.evalshift.dev, or your--host), only when you runlogin,whoami,push, orcompare --push. The local commands —doctor,run,evaluate,analyze,report,bundle— send nothing to EvalShift-operated services.
## What push sends, block by block
push uploads one file, run_bundle.json.gz, plus request metadata: the bearer token (an Authorization header, sent only to the configured host), and, on the request that creates the run, the compressed bundle size and a copy of the bundle's manifest block. login additionally sends a client name that includes your machine's hostname, so you can recognize the session in the dashboard. The bundle itself contains:
| Block | What is inside |
|---|---|
| manifest | Run id, org/project slug, source and target model ids, suite name, git commit SHA, branch name, PR number, the local suite file path as a string (it can reveal directory or user names), two content hashes, the run timestamp, and the CLI version. |
| examples[] | One row per prompt × example: the example's template variables (inputs) verbatim; its expected reference output verbatim; both models' full output text; the replay's own tool-call traces (tool names, arguments and call ids, each tagged with its round; final text and refusal messages; capped at 256 KB per side with a truncated flag — no tool results, no model-call events); per-evaluator scores and error strings; per-side cost and latency; tags and slice names. Agent traces imported with evalshift traces import are not included. |
| aggregate · analysis · decision · economics | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. |
| methodology_notes | The model ids and the statistical-contract sentences shown in every report. |
| insights | The machine-written run narrative, when one was generated. It is prose about your run and can paraphrase or quote the regressions it summarizes. |
| evaluator_config | Config version; prompt list metadata only — prompt names, file paths, and variable names, with every prompt body replaced by a content_hash; defaults (model ids, concurrency, cache flag, cost ceiling, max_tokens, samples_per_example); an always-empty slices list, kept only so the config hash stays stable; and the full evaluators block, including each llm_judge criterion_prompt text — keep judge criteria free of secrets. |
| dataset_snapshot | Suite path, example count, slice names, and one examples_hash. No example content. |
## What never leaves your machine
- +Provider API keys and the Cloud token. Neither is ever inside a bundle. Keys go only to their own providers; the token goes only to the configured host as an auth header.
- +Prompt bodies and system prompts. A
manualprompt's content is replaced by acontent_hash; apython_stringprompt's body never enters the config at all — only its file path and variable name do. - +Suite conversation histories (
history, including any embedded system message). The dataset snapshot ships hashes, not examples. - +Tool definitions. Toolsets — names, descriptions, JSON schemas — are not in the bundle; only the calls a model actually made at run time appear, in the traces.
- +Imported agent traces. Timelines attached with
evalshift traces import— tool results, retrievals, guardrail verdicts — stay in the local run directory. They feed theagent_traceevaluator, whose scores do upload, but the timelines themselves never enter the bundle. - +Local artefacts:
raw.jsonl(the raw provider requests and responses), the SQLite response cache,.evalshift/captures/,state.json,report.json, andreport.html.
The content hashes that replace this data (dataset_hash, examples_hash, prompts[].content_hash) are SHA-256 digests, so Cloud diffs and baselines still align across runs without the content itself uploading.
inputs, expected, both model outputs, and tool traces upload verbatim. If your suite rows contain customer data, or a model echoes a secret it was given at run time, that content is in the bundle — EvalShift cannot tell the difference. Redact at capture time with the SDK redaction boundary so sensitive values never reach disk, and inspect the bundle before pushing.## Inspect before you push
bundle builds the artefact without uploading, and push --bundle uploads exactly the file you inspected — the bytes are deterministic, so what you read is what is sent:
# build the exact artefact a push would upload — nothing is sent evalshift bundle <run-id> # read every byte of it gunzip -c .evalshift/runs/<run-id>/run_bundle.json.gz | jq . | less # happy with it? push exactly that file evalshift push --bundle .evalshift/runs/<run-id>/run_bundle.json.gz
If a run must not leave the machine, simply never push it — every local artefact, the HTML report included, works without an account. See Cloud setup for the push workflow itself, and the CLI reference for AI tools ↗ for the same contract in machine-readable form.