Add your agent

Package your agent as a container image and register it as a candidate on your bench.

When you finish this page, your agent is a candidate on your bench, ready to run in a trial. The samples use the shell variables from Before you start.

Three ways an agent can join a bench

Hosted endpoint. You run the agent and expose it at a URL. The bench talks to it over http, WebSocket or voice WebSocket, with the same field mapping as Your image. For the agent to use the bench’s twins, Veris sets up a twin sandbox of the bench’s world; you point your agent at it and give the bench the sandbox’s id (Pinned twin sandbox in the console). Write to hello@veris.ai to set up the twin sandbox.

Managed harness. See the stock agents Veris can add to your bench in the Library , and write to hello@veris.ai to add one.

The rest of this page is about Your image.

Let your coding agent do it

Most of the work is wiring your agent to the bench, and your coding agent can do it for you with the add-your-agent skill. Install it:

/plugin marketplace add veris-ai/plugins
/plugin install bench@veris

codex plugin marketplace add veris-ai/plugins
codex plugin add bench@veris

npx skills add veris-ai/plugins --skill add-your-agent

Have these ready. The skill asks for any that are missing:

• your API key in VERIS_API_KEY (see Before you start) and your bench’s id;
• a registry Docker can push to, such as ghcr.io/<you>;
• the keys your agent needs, such as OPENAI_API_KEY. They go in the bench, not in the image.

The skill adds a server file, a Dockerfile and bench/candidate.json beside your agent, and leaves your agent’s own code as it is where it can. It builds and pushes the image, registers the candidate, and runs a smoke trial. It is done when every smoke attempt passes the checks in step 8, and it ends with the candidate’s id and a link to the trial.

Veris helps you add your first agent: write to hello@veris.ai.

Under the hood

The skill follows the steps below. Read them to see what changes in your repository, or to do each step yourself.

1. Answer over http or ws

Your container runs a server. The bench’s simulated user talks to it, one turn at a time, over http or WebSocket (ws). There is no Veris SDK and no base image.

http. Each user turn is one request. By default the bench sends:

POST /chat
{"message": "Hi, I need to change my delivery address.", "session_id": "a1b2c3"}

and expects back:

200
{"response": "Sure. What is your order number?", "session_id": "a1b2c3"}

The first request has no session_id. Return one, and every later request in that conversation carries it back. You choose the path and the field names; see http field mapping.

Any language or framework works, as long as the server answers this shape. If your agent has no server yet, for example it runs in a terminal or inside another app, add a small server that calls the function your agent runs for each turn. Two examples of a minimal agent:

import os
from collections import defaultdict
 
from fastapi import FastAPI
from openai import AsyncOpenAI
from pydantic import BaseModel
 
app = FastAPI()
client = AsyncOpenAI()  # reads OPENAI_API_KEY
history: dict[str, list[dict]] = defaultdict(list)
 
 
class Turn(BaseModel):
    message: str
    session_id: str | None = None
 
 
@app.get("/health")
async def health():
    return {"status": "ok"}
 
 
@app.post("/chat")
async def chat(turn: Turn):
    session_id = turn.session_id or os.urandom(8).hex()
    messages = history[session_id]
    messages.append({"role": "user", "content": turn.message})
    reply = await client.chat.completions.create(
        model=os.environ.get("MODEL", "gpt-4o-mini"), messages=messages
    )
    text = reply.choices[0].message.content or ""
    messages.append({"role": "assistant", "content": text})
    return {"response": text, "session_id": session_id}

FROM python:3.12-slim
RUN pip install --no-cache-dir "fastapi~=0.115" "uvicorn[standard]~=0.32" "openai~=1.58"
WORKDIR /app
COPY app.py .
RUN useradd --uid 10001 agent
USER agent
EXPOSE 8008
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8008"]

import http from "node:http";
import { randomUUID } from "node:crypto";
import OpenAI from "openai";
 
const client = new OpenAI(); // reads OPENAI_API_KEY
const history = new Map();
 
async function readJson(req) {
  let body = "";
  for await (const chunk of req) body += chunk;
  return JSON.parse(body);
}
 
function send(res, status, payload) {
  res.writeHead(status, { "Content-Type": "application/json" });
  res.end(JSON.stringify(payload));
}
 
http
  .createServer(async (req, res) => {
    if (req.method === "GET" && req.url === "/health") return send(res, 200, { status: "ok" });
    if (req.method !== "POST" || req.url !== "/chat") return send(res, 404, { error: "not found" });
    try {
      const turn = await readJson(req);
      const sessionId = turn.session_id ?? randomUUID();
      const messages = history.get(sessionId) ?? [];
      messages.push({ role: "user", content: turn.message });
      const reply = await client.chat.completions.create({
        model: process.env.MODEL ?? "gpt-4o-mini",
        messages,
      });
      const text = reply.choices[0].message.content ?? "";
      messages.push({ role: "assistant", content: text });
      history.set(sessionId, messages);
      send(res, 200, { response: text, session_id: sessionId });
    } catch (err) {
      console.error(err);
      send(res, 500, { error: String(err) });
    }
  })
  .listen(8008, "0.0.0.0");

{
  "type": "module",
  "private": true,
  "dependencies": { "openai": "^5.23.2" }
}

FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm install --omit=dev
COPY server.mjs .
RUN chown -R node:node /app
USER node
EXPOSE 8008
CMD ["node", "server.mjs"]

ws. One text frame in is one user turn. Send the reply as one text frame, or stream it without pausing for more than about one second: a longer pause is read as the end of the reply. Keep conversation state per connection; the bench opens one connection per attempt.

Whichever you pick, your server must:

• listen on a port of 1024 or above, as a non-root user (the container gets no extra Linux privileges);
• answer within 600 seconds of the container starting;
• serve a health path (such as /health) if it accepts connections before it can answer, for example while it loads a model;
• log to stdout and stderr. A failed attempt shows the last lines.

2. Reach the world’s services

Each task runs in a world of twins: working copies of the services your agent calls. A bench reaches them in one of two ways. Benches use direct unless Veris turns the gateway on for yours.

Through the Veris gateway. Your agent calls vendors as it does in production, at their usual addresses such as https://api.stripe.com. The gateway sends those calls to the twins. Nothing in your image changes.

Direct. Your agent reads each twin’s address from an environment variable and points its client there:

• each twin sets the variable its vendor’s clients usually read, for example STRIPE_API_BASE for Stripe, GOOGLE_CALENDAR_API_BASE for Google Calendar, SALESFORCE_INSTANCE_URL for Salesforce and DATABASE_URL for Postgres;

• VERIS_DS_SERVICE_URLS holds every twin’s name and address as JSON, for example {"postgres": "postgresql://..."}. If your agent reads its own setting, such as BANK_DB_URL, copy the twin’s address into it at startup. The twins’ names are the services of your bench’s world:

  curl -s "$BENCH_API/benches/$BENCH_ID/worlds" \
    -H "Authorization: Bearer $VERIS_API_KEY" | jq '.items[] | [.services[].service]'

On a direct bench, an agent with https://api.stripe.com written into its code reaches the real Stripe, not the twin. Databases such as Postgres always come through their variable, in both modes.

Node and Postgres. The Postgres twin’s address carries sslmode=require, and its certificate is self-signed. node-postgres (pg) 8.16 and later treat require as full verification, so every query fails. Add uselibpqcompat=true to the address’s query, or pass ssl: { rejectUnauthorized: false }. Python’s psycopg connects as it is.

3. Let the grader see your model calls

Veris records your agent’s model calls, including tool calls and their results, from inside your process. Nothing is proxied. The grader reads them beside the transcript, so an outcome such as “the refund was issued” can pass on the tool result that shows it. Without them, the grader judges the transcript alone.

This works when:

• your agent is a Python or Node process (Node 18.19+ or 20.6+);
• it calls its model through a supported SDK. Python: OpenAI, OpenAI Agents, Anthropic, Mistral, Google GenAI, LangChain. Node: OpenAI (the openai package 5 or later; with 4.x nothing is recorded), Anthropic, Claude Agent SDK, LangChain, AWS Bedrock, and the Vercel AI SDK with experimental_telemetry: { isEnabled: true };
• its tools are model function tools;
• any PYTHONPATH or NODE_OPTIONS you need is set in image.env (step 7), not only in the Dockerfile;
• you don’t set OTEL_* variables yourself;
• for Node, the working directory is your package’s directory (where node_modules is), and the user that runs the agent can write to it. If it can’t, nothing is recorded and nothing says so.

To check, read the attempt (step 8): agent_trace_reason is null, and its agent_trace.json has llm_calls above 0. A Node agent’s log also has a line veris-otel-node: instrumented <SDK> for each SDK that is recorded.

4. Build and push the image

Build for linux/amd64, push the image to a registry, and refer to it by digest:

docker buildx build --platform linux/amd64 -t ghcr.io/<you>/my-agent:1.0 --push .
docker buildx imagetools inspect ghcr.io/<you>/my-agent:1.0 \
  --format '{{json .Manifest}}' | jq -r .digest   # prints sha256:...

• Attempts run on linux/amd64. On an Apple Silicon Mac, a plain docker build makes an arm64 image that can’t start there.
• The reference needs a tag or a digest. A bare name or :latest is refused.
• Use a digest (ghcr.io/<you>/my-agent@sha256:...) for anything you compare. The bench pulls the image fresh for every attempt, so a tag that moves changes the candidate under you. Two digests of one repository are how you compare two versions of your agent.
• Public GitHub Container Registry and Google Artifact Registry images work as they are. Docker Hub limits anonymous pulls, and a limited pull fails the attempt; prefer the other two.

Check that the bench can pull your image:

curl -s -X POST "$BENCH_API/registries/check-image" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"ref": "ghcr.io/<you>/my-agent@sha256:..."}'

{"status": "public", "detail": "public image, no login needed", "registry_id": null, "prefix": null, "digest": "sha256:...", ...}

5. Private registry: save a login

If your image is private, save a registry login once for your workspace. Every bench in the workspace pulls with it.

curl -s -X PUT "$BENCH_API/registries" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"kind": "github", "prefix": "ghcr.io/<you>", "username": "<you>", "secret": "<token>"}'

• kind is one of dockerhub, github, gitlab, azure, ecr, gar, other.
• prefix is the part of the image reference the login covers. A second login with the same prefix replaces the first.
• For github, use a classic personal access token with only the read:packages scope. A token with more scopes is refused.
• For ecr, username is the access key id, secret the secret access key, and region is required. For gar, secret is the service account key JSON and username can be left out.
• Veris signs in to the registry before saving. A login that doesn’t work is refused with the registry’s own error.

Then check the image again. The API’s check-image should say ok and name the login:

{"status": "ok", "detail": "pulls with the ghcr.io/<you> login", "registry_id": "regcred_...", "prefix": "ghcr.io/<you>", "digest": "sha256:...", ...}

6. Keep secrets in the bench

Put API keys and other secrets in the bench’s candidate-env. Every candidate on the bench gets them, and neither the console nor the API shows a secret’s value again.

curl -s -X PUT "$BENCH_API/benches/$BENCH_ID/candidate-env/OPENAI_API_KEY" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"value": "sk-...", "is_secret": true}'

{"id": "cenv_...", "bench_id": "bench_...", "key": "OPENAI_API_KEY", "is_secret": true, "value": null, ...}

Plain settings for one candidate, such as MODEL or a feature flag, go in image.env instead (step 7). Anyone who can read the bench can read image.env. Names that start with VERIS_ are refused in both places. The candidate-env also refuses names that Veris’s own agents use, such as MODEL and SYSTEM_PROMPT; set those in image.env.

7. Register the candidate

curl -s -X POST "$BENCH_API/benches/$BENCH_ID/candidates" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d @candidate.json

{
  "name": "my-agent 1.0",
  "image": {
    "ref": "ghcr.io/<you>/my-agent@sha256:...",
    "port": 8008,
    "path": "/chat",
    "health_path": "/health",
    "working_dir": "/app",
    "env": {"MODEL": "gpt-4o-mini"}
  },
  "channel_config": {
    "init": null,
    "channel": {
      "type": "http",
      "url": null,
      "method": "POST",
      "request": {"message_field": "message", "session_field": "session_id"},
      "response": {"type": "json", "message_field": "response", "session_field": "session_id"}
    }
  }
}

{
  "id": "cand_...",
  "bench_id": "bench_...",
  "name": "my-agent 1.0",
  "image": {"ref": "ghcr.io/<you>/my-agent@sha256:...", "port": 8008, "path": "/chat", "health_path": "/health", ...},
  "channel_config": {...},
  ...
}

8. Check that it works

Start a smoke trial with your candidate, as in Run a trial, and wait for it. An attempt that completed doesn’t yet show that your agent works on the bench: an agent whose every database call fails still completes. Check each attempt:

This prints those for each attempt. llm_calls comes from the attempt’s agent_trace.json; reached asks each twin’s ledger for a request from your agent.

curl -s "$BENCH_API/trials/$TRIAL_ID/attempts?candidate_id=$CANDIDATE_ID&limit=50" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  | jq -r '.items[].id' | while read -r a; do
    att=$(curl -s "$BENCH_API/attempts/$a" -H "Authorization: Bearer $VERIS_API_KEY")
    llm=$(curl -s "$BENCH_API/attempts/$a/artifacts/agent_trace.json" \
      -H "Authorization: Bearer $VERIS_API_KEY" | jq '.llm_calls // 0' 2>/dev/null)
    twins=""
    for s in $(jq -r '.ledger_summary.sources[]? | select(.kind == "service") | .name' <<<"$att"); do
      n=$(curl -s "$BENCH_API/attempts/$a/ledger?plane=vendor&source=$s&limit=1" \
        -H "Authorization: Bearer $VERIS_API_KEY" | jq '.events | length')
      twins="$twins$s:$n "
    done
    jq -c --argjson llm "${llm:-0}" --arg twins "$twins" \
      '{id, status, turns, llm_calls: $llm, tool_calls, agent_trace_reason, reached: $twins, passed}' <<<"$att"
  done

{"id": "att_...", "status": "completed", "turns": 6, "llm_calls": 6, "tool_calls": 7, "agent_trace_reason": null, "reached": "postgres:1 ", "passed": true}

agent_trace_reason is judge_disabled_trace when your bench’s grader is set not to read traces; then skip that check.

Whether an attempt passed is about your agent, not about this setup. Two attempts say little; a full trial does.

Next: run a trial.

Troubleshooting

When an attempt fails, its error names the cause and ends with the last lines your container logged. Read it in the console’s attempt view, or with GET $BENCH_API/attempts/<attempt id>.

Each attempt also keeps artifacts. List them with jq '.artifact_uris | keys' on the attempt, and read one with GET $BENCH_API/attempts/<attempt id>/artifacts/<name>:

Why an attempt didn’t pass is in its assertion_results: each check’s statement, whether it passed, and the grader’s reasoning.

API error messages call both Your image and Managed harness candidates “managed”, because Veris runs them.

http field mapping

Set these on channel_config.channel when your API doesn’t use the default shape.

For example, an agent that replies {"answer": "...", "conversation": "..."} needs "request": {"session_field": "conversation"} and "response": {"message_field": "answer", "session_field": "conversation"}.