Run a trial

Start a trial, wait for it, read the pass rate, and compare two versions of your agent.

When you finish this page, you have a pass rate for your agent and know how to compare a new version against it on the same tasks. You need a candidate id from Add your agent and the shell variables from Before you start.

How a trial runs

• A trial fixes the bench’s tasks at the moment it starts.
• An attempt is one candidate on one task, with one archetype (a kind of simulated user the bench may define), once.
• Repeats run each pairing more than once: 1 by default for a smoke trial, 3 for a full one.

So each candidate gets tasks × archetypes × repeats attempts. Each one runs in a fresh world and is graded on its own.

1. Start a smoke trial

Start small, to check that your agent works on the bench: a smoke trial on two tasks, with no archetypes.

curl -s "$BENCH_API/benches/$BENCH_ID/tasks?limit=200" \
  -H "Authorization: Bearer $VERIS_API_KEY" | jq -r '.items[] | "\(.id)  \(.name)"'

curl -s -X POST "$BENCH_API/benches/$BENCH_ID/trials" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-agent smoke",
    "kind": "smoke",
    "candidate_ids": ["cand_..."],
    "task_ids": ["task_...", "task_..."],
    "persona_ids": [],
    "repeats": 1
  }'

{
  "id": "trial_...",
  "name": "my-agent smoke",
  "kind": "smoke",
  "status": "running",
  "candidate_ids": ["cand_..."],
  "task_ids": ["task_...", "task_..."],
  "persona_ids": [],
  "repeats": 1,
  "attempt_count": 2,
  ...
}

2. Wait for it

export TRIAL_ID=trial_...
export CANDIDATE_ID=cand_...
 
while :; do
  left=$(curl -s "$BENCH_API/trials/$TRIAL_ID/results" \
    -H "Authorization: Bearer $VERIS_API_KEY" \
    | jq --arg c "$CANDIDATE_ID" '.rollups[] | select(.candidate_id == $c) | .pending')
  echo "$(date +%T) attempts still running or being graded: $left"
  [ "$left" = 0 ] && break
  sleep 60
done

Each attempt ends as completed (it ran; passed says how it did) or failed (it could not finish). How long a trial takes depends on the bench and your agent: a small smoke trial can finish in minutes, and a full trial with many tasks and three repeats can take hours.

3. Read the result

curl -s "$BENCH_API/trials/$TRIAL_ID/results" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  | jq '.rollups[] | {name, n, passes, pending, pass_rate, ci_low, ci_high, cost_per_task}'

{
  "name": "my-agent 1.0",
  "n": 2,
  "passes": 1,
  "pending": 0,
  "pass_rate": 50.0,
  "ci_low": 9.45,
  "ci_high": 90.55,
  "cost_per_task": 0.0124
}

curl -s "$BENCH_API/trials/$TRIAL_ID/attempts?candidate_id=$CANDIDATE_ID&limit=200" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  | jq '.items[] | {id, task_id, status, passed, failure_class, error}'

{"id": "att_...", "task_id": "task_...", "status": "completed", "passed": false, "failure_class": null, "error": null}
{"id": "att_...", "task_id": "task_...", "status": "completed", "passed": true, "failure_class": null, "error": null}

Two attempts say little: the interval runs from 9% to 91%. Run a full trial before you draw conclusions.

When an attempt fails on Veris’s side

An attempt that failed because of Veris, not your agent, has "failure_class": "platform" ("candidate" means your agent’s side). You can leave such attempts out of the pass rate. This is API only:

curl -s -X PUT "$BENCH_API/trials/$TRIAL_ID/exclusions" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"attempt_ids": ["att_..."], "reason": "platform failure"}'

Send the same ids with "reason": null to count them again.

4. Compare a new version

A candidate is one fixed image. A new version of your agent is a new candidate: push the new image, register it, then add it to the same trial. Both versions then face the same tasks, the same archetypes and the same worlds.

curl -s -X POST "$BENCH_API/trials/$TRIAL_ID/candidates" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"candidate_ids": ["cand_..."]}'

{"name": "my-agent 1.1", "n": 2, "passes": 1, "pending": 0, "pass_rate": 50.0, "ci_low": 9.45, "ci_high": 90.55, "cost_per_task": 0.0080}
{"name": "my-agent 1.0", "n": 2, "passes": 1, "pending": 0, "pass_rate": 50.0, "ci_low": 9.45, "ci_high": 90.55, "cost_per_task": 0.0124}

Reading a change. Compare the intervals, not just the pass rates. When two intervals overlap, the difference may be noise. With 25 attempts, an interval is about ±18 points wide; more tasks or more repeats narrow it.

If you changed a candidate after it ran, re-run it in place with {"candidate_ids": ["cand_..."], "replace": true}.

When the bench changes under a trial

A trial keeps its tasks fixed. When the bench changes, read the trial to see whether a new candidate still gets a fair comparison:

curl -s "$BENCH_API/trials/$TRIAL_ID" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  | jq '{outdated, outdated_reasons, edited_task_ids, stale_persona_ids}'

• outdated: true, with outdated_reasons: the bench changed since the trial started, for example a task was removed. The trial refuses new candidates.
• edited_task_ids or stale_persona_ids not empty: tasks or archetypes were edited since they ran. The trial still accepts new candidates, but their attempts run the edited text, so they no longer face the same tasks as the earlier versions.

In both cases, start a new trial and compare versions there.

Next: run it from CI/CD.