Skip to Content
Core ConceptsSimulations & Runs

Simulations & Runs

What is a Simulation?

A simulation is a single test execution — one scenario running in one isolated container. During a simulation:

  1. A fresh container starts with your agent and configured mock services
  2. Mock services seed their data based on the scenario context
  3. The actor (simulated user) initiates the conversation
  4. Your agent responds, potentially calling mock services
  5. The actor evaluates responses and continues pursuing its objectives
  6. The simulation ends when objectives are met or max_turns is reached

Each simulation produces a transcript — a complete record of all messages, API calls, actor reasoning, and service interactions.

What is a Run?

A run is a collection of simulations. When you create a run, every scenario in the selected scenario set becomes one simulation. Runs support concurrency — multiple simulations execute in parallel for faster results.

Creating a Run

Interactive mode

veris run create

The CLI prompts you to select an environment, scenario set, and concurrency level.

With flags

veris run create \
  --scenario-set-id ss_abc123 \
  --concurrency 10 \
  --simulation-timeout 300
  • --concurrency — Maximum parallel simulations (default: 10)
  • --simulation-timeout — Timeout per simulation in seconds (60–3600)

Monitoring Progress

From the CLI

# Check status (one-time)
veris run status RUN_ID
 
# Watch mode (polls every 3 seconds)
veris run status RUN_ID --watch
 
# Stream events
veris run logs RUN_ID --follow

From the Console

Navigate to Simulations and click on your run. The detail page shows:

  • Overall progress bar
  • Per-simulation status (pending, running, completed, failed)
  • Duration and timing information

Click an individual simulation to view its full transcript.

Viewing Results

# List all simulations in a run
veris simulations get RUN_ID
 
# View a specific simulation's result
veris simulations get RUN_ID SIM_ID

In the Console, the simulation detail page shows:

  • The complete conversation transcript (all messages between actor and agent)
  • Actor’s internal reasoning and task board updates
  • All API calls made to mock services with request/response payloads
  • Events and timing

Local Runs

For faster iteration during development, run simulations locally without pushing to the cloud:

# Run all scenarios in the scenarios/ directory
veris run local
 
# Run specific scenarios
veris run local schedule_meeting cancel_event
 
# Options
veris run local --skip-build           # Don't rebuild Docker image
veris run local --concurrency 3        # Limit parallel containers
veris run local --scenarios-dir ./tests # Custom scenarios path

Local runs use Docker on your machine and load environment variables from a .env file in your project root. They don’t require pushing to the Veris registry first.

Local run output

Each simulation creates a log directory at .veris/logs/<simulation_id>/ with the complete container output. A summary table shows the scenario, simulation ID, exit code, and log path.

Cancelling a Run

veris run cancel RUN_ID

This stops all pending and running simulations in the run. You’ll be asked to confirm before cancellation proceeds.

CLI Commands

# Create a run
veris run create [--scenario-set-id ID] [--concurrency N]
 
# List runs
veris run list [--status STATUS] [--env-id ID]
 
# Check run status
veris run status RUN_ID [--watch]
 
# Stream run events
veris run logs RUN_ID [--follow]
 
# Cancel a run
veris run cancel RUN_ID
 
# Run locally
veris run local [SCENARIO...] [--skip-build] [--concurrency N]
 
# View simulation results
veris simulations get RUN_ID [SIM_ID]
Scenarios & GradersEvaluations