EvalShift Cloud stores pushed runs, diffs them across branches, and comments on PRs — useful once a team wants shared reports, baselines, and PR gating. It is strictly opt-in: the CLI works fully without an account, model calls always run locally with your own provider keys, and nothing leaves your machine unless you run push (or compare --push). Provider API keys are never uploaded.

## 1. Sign up and create a token

Open EvalShift Cloud and create an account. On your own machine, evalshift login (below) issues a personal token — the right credential for a workstation, and the wrong one for CI, because it stops working when your membership does.

For CI, mint a service-account key under Settings → API tokens → Service accounts (/app/<org-slug>/settings/tokens): role member (a viewer cannot upload a run), scopes run:create, run:read, and policy:read (the last is what lets the default fail-on: policy gate read the verdict; without it the check falls back to fail-on: regression silently). A service account is member or viewer, never owner or admin, so it cannot auto-create a project — create the project once in the web app (or with a first push from your own owner or admin login) and set the Action’s create-project: false.

## 2. Authenticate the CLI

evalshift login                       # device-code browser flow
evalshift login --token es_...        # or paste a token (verified via GET /me)
evalshift whoami

Credentials live in ~/.evalshift/credentials (owner-only permissions). Precedence: CLI flags (--host/--token) > env (EVALSHIFT_HOST/EVALSHIFT_TOKEN) > credentials file. --no-browser prints the approval URL for remote shells.

## 3. Point config at a project

Projects are org/project slugs — from --project, or the project: key in config. Missing projects are auto-created when the token has owner or admin access to the org. Add the project path and the migration policy the gate enforces — it travels with every pushed run:

version: 1
project: acme/customer-router
migration_policy:
  max_overall_regression_rate: 0.03
  max_critical_regressions: 0
  min_equivalence_rate: 0.95

## 4. Push the first Cloud run

compare --push runs the local pipeline, builds run_bundle.json.gz, uploads it, finalizes it, and prints the Cloud run URL. Pushes are idempotent per project on the local run id, and that URL carries the server-minted id instead. evalshift bundle builds the artefact without uploading if you want to inspect it first.

evalshift compare --yes --push

# or push an existing local run
evalshift push <run-id>

## 5. Install the GitHub Action

The minimal step is below; evalshift init --ci writes a fuller, per-suite version of it — a discover job, one job per suite selected by suite-name, and an evalshift gate join job to require in branch protection. The Action checks the plan, pushes the candidate run, finds a compatible baseline on the PR base branch, asks for the policy verdict, posts or updates one PR comment, and sets the evalshift/regression commit status. Add EVALSHIFT_TOKEN plus your provider key (e.g. GEMINI_API_KEY) as GitHub repository secrets. Full inputs, gating modes, and recipes live in the GitHub Action section.

permissions:
  contents: read
  pull-requests: write
  issues: write
  statuses: write

steps:
  - uses: actions/checkout@v7
  - uses: evalshift/evalshift-action@v0
    with:
      token: ${{ secrets.EVALSHIFT_TOKEN }}
      suite-name: support_agent       # a suites: key from evalshift.yaml
      evalshift-version: "1.1.0"      # the CLI version you run locally
      fail-on: policy                 # the default

## 6. Review diffs and the policy verdict

Open the run URL to inspect the report. When a compatible baseline exists, the PR comment links to a Cloud diff with aggregate, slice, and per-example deltas. Under the default fail-on: policy the Action asks EvalShift Cloud for the run’s verdict against the migration policy it was pushed with (GET /runs/{id}/policy-check) and fails the job only on fail; if that check is unreachable it falls back to regression and says so. fail-on: regression instead fails when the Cloud diff reports regressed examples. See Gating & PR feedback.

## Troubleshooting

• +401 from the CLI or Action means the token is missing, revoked, or expired, or the host is wrong.
• +403 means the token authenticated but lacks a permission — the CLI and Action name the missing key (e.g. run:create).
• +404 on an org or project means the token can’t see it: a wrong slug, or a token scoped to a different org or project. Existence is hidden rather than refused.
• +No diff link means no compatible baseline exists yet on the base branch. Push one run on main, then open another PR.
• +Browser run viewing needs storage CORS to allow the web origin.