CI/CD Integration

Run simulations on every pull request and post results as a PR comment.

How It Works

The veris ci run command wraps the full simulation-to-evaluation pipeline into a single command:

Creates a simulation run against your scenario set
Polls until all simulations complete
Auto-resolves the grader for your scenario set
Triggers evaluation and polls until complete
Outputs a markdown summary to stdout

Progress logs go to stderr, clean markdown goes to stdout — so you can redirect the report to a file while still seeing progress in CI logs.

Setup

Configure CI locally

Run veris ci run interactively once to select a scenario set. This saves the config to .veris/config.yaml:


veris ci run

You’ll be prompted to select a scenario set. The selection is saved so future runs need zero flags.

Commit the config

Make sure .veris/config.yaml is not gitignored, then commit it:


git add .veris/config.yaml
git commit -m "add veris ci config"

If your .gitignore excludes .veris/config.yaml, add !.veris/config.yaml to un-ignore it. The config file only contains environment and scenario set IDs — no secrets.

Add your API key as a secret

In your GitHub repo, go to Settings > Secrets and variables > Actions and add:

VERIS_API_KEY — your Veris API key

Create a GitHub Actions workflow

.github/workflows/veris.yaml


name: Veris Simulation
on:
  pull_request:
    branches: [main]
 
permissions:
  contents: read
  pull-requests: write
 
jobs:
  simulate:
    runs-on: ubuntu-latest
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4
 
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
 
      - name: Install veris-cli
        run: pip install veris-cli
 
      - name: Build & push agent image
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris env push --tag ${{ github.sha }} --remote
 
      - name: Run simulation & evaluation
        run: |
          veris ci run --image-tag ${{ github.sha }} > veris-summary.md
 
      - name: Comment on PR
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: veris-summary.md

CLI Reference


# Everything from config — zero flags
veris ci run
 
# Override image tag (common in CI)
veris ci run --image-tag $(git rev-parse --short HEAD)
 
# Override everything
veris ci run --scenario-set-id X --env-id Y --concurrency 5

Flag	Description	Default
`--scenario-set-id`	Scenario set to run	From `.veris/config.yaml`
`--env-id`	Environment ID	From `.veris/config.yaml`
`--concurrency`	Parallel simulation jobs	`10`
`--image-tag`	Docker image tag to use	`latest`
`--simulation-timeout`	Timeout per simulation (seconds)	Server default

Output Format

The markdown summary written to stdout includes:

Run metadata — run ID, status, scenario set, duration
Grading results — per-scenario scores (if graders are configured)
Assertion results — pass/fail verdicts with criteria counts (if assertions are configured)