Veris Bench

Put your agent on a bench, see how each version scores, and run it from CI.

Veris Bench runs your agent on realistic tasks in simulated worlds and grades every attempt. These pages take you from an agent in a repository to a pass rate for each version of it, and to a CI job that runs the bench for you.

Terms

• Bench: a set of tasks for one kind of job, with everything needed to run them.
• World: the systems your agent works in during a task. Each service in it is a twin: a copy of a real API, such as a card processor or a database, that behaves like the real one and keeps state.
• Candidate: one fixed version of an agent on the bench. A new version of your agent is a new candidate.
• Trial: one run of the bench. Each candidate meets each task in an attempt. With repeats, each task runs more than once.
• Pass rate: the share of attempts that passed, with a 95% confidence interval. The interval tells you how much of a change is noise.

Before you start

You can do every step in the console  or with the API. Each step shows both: pick Console or API, and the pages remember your choice. The API samples are plain curl, so you can also hand them to your coding agent.

For the API you need a key. Issuing one needs a person in the console: open Overview and choose Issue on the API keys card. Name the key, for example github-actions, and choose Issue key. The console shows the key once, so copy it then. Keys start with vbk_.

Every API sample on these pages uses the same shell variables. Set them once:

export BENCH_API=https://bench.api.veris.ai/v1
export VERIS_API_KEY=vbk_...   # your key

Every call sends the key as Authorization: Bearer $VERIS_API_KEY.

Get a bench into your workspace

Adding an agent and starting trials change the bench, so you need two things:

• a bench owned by your own workspace, and
• the member role in that workspace. A viewer can read, but not change anything.

Veris may already have built a bench in your workspace. If not, start from a template:

# List the templates
curl -s "$BENCH_API/benches?shared=true" \
  -H "Authorization: Bearer $VERIS_API_KEY"
 
# Fork one into your workspace
curl -s -X POST "$BENCH_API/benches" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-bench", "source": {"type": "fork", "bench_id": "<template id>"}}'

export BENCH_ID=bench_...   # your bench id
 
curl -s "$BENCH_API/benches/$BENCH_ID" \
  -H "Authorization: Bearer $VERIS_API_KEY"

{
  "id": "bench_...",
  "name": "my-bench",
  "access": "write",
  ...
}

A fork is a one-time copy that you own. It brings the tasks, the world, the simulated users, any candidates and the settings. It does not bring secrets or past trials, so you add your own keys again when you add your agent.

If you can’t change the bench (a change is refused, or the API answers "access": "read"), there are three causes:

• You are a viewer in your workspace: ask your workspace owner for the member role.
• The bench is a template: fork it, as above.
• The bench belongs to another workspace and is shared with yours for viewing only. It can’t be forked: ask Veris for a copy in your workspace.

Next

1. Add your agent: package it, push it, register it as a candidate. Your coding agent can do this for you with the add-your-agent skill.
2. Run a trial: start a trial, wait for it, read the pass rate, compare versions.
3. Regression testing in CI/CD: do the same from GitHub Actions.

Find more

• The console  shows your benches, trials and every attempt’s transcript.
• The card-replacement agent  in the Veris cookbook is a full worked example. It runs on a bench every night.
• Questions: hello@veris.ai.
• Veris for coding agents is a different product: twins your coding agent builds and tests against.