Fireworks AI

Run any agent on any Fireworks AI model, on your own Fireworks key.

Fireworks AI  serves open models through an OpenAI-compatible API; see the Fireworks model library . On Veris Bench, any agent can run on them: your own agent with its own harness, or an open-source harness such as Pi, Hermes Agent or OpenCode.

The API samples use the shell variables from Before you start.

1. Add your Fireworks key

Create an API key in your Fireworks account , then add it to your workspace. Every harness Veris runs for you can use it. If you bring your own agent, add the key to your bench’s candidate-env too.

curl -s -X PUT "$BENCH_API/workspace-keys/fireworks" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api_key": "fw_..."}'

curl -s -X PUT "$BENCH_API/benches/$BENCH_ID/candidate-env/FIREWORKS_API_KEY" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"value": "fw_...", "is_secret": true}'

Attempts bill your Fireworks account at its own rates.

2. Your own agent on Fireworks

Point your agent’s model client at Fireworks. With the OpenAI SDK, that is the base URL and the key:

import os
 
from openai import AsyncOpenAI
 
client = AsyncOpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.environ["FIREWORKS_API_KEY"],
)
 
reply = await client.chat.completions.create(
    model=os.environ["MODEL"],  # e.g. accounts/fireworks/models/kimi-k3
    messages=messages,
)

import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: "https://api.fireworks.ai/inference/v1",
  apiKey: process.env.FIREWORKS_API_KEY,
});
 
const reply = await client.chat.completions.create({
  model: process.env.MODEL, // e.g. accounts/fireworks/models/kimi-k3
  messages,
});

Any other framework or harness that takes an OpenAI-compatible base URL works the same way.

Set the model per candidate in image.env, such as {"MODEL": "accounts/fireworks/models/kimi-k3"}. One image registered as several candidates, each with its own MODEL, compares models under the same agent.

The rest is the same as for any agent: Add your agent covers building, pushing and registering it. Model calls made through a supported SDK are recorded whatever endpoint it points at, so the grader sees your agent’s tool calls on Fireworks too (step 3).

3. An open-source harness on Fireworks

Veris also runs open-source harnesses, such as Pi, Hermes Agent and OpenCode, on the Fireworks model you choose. Write to hello@veris.ai with your bench’s id, the harness, and the Fireworks model id for each candidate, and Veris adds them to your bench.

4. Compare them in one trial

Put the candidates in one trial, for example your own agent on two Fireworks models and an open-source harness on a third. Every candidate meets every task.

curl -s -X POST "$BENCH_API/benches/$BENCH_ID/trials" \
  -H "Authorization: Bearer $VERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "fireworks models", "kind": "full", "candidate_ids": ["cand_...", "cand_...", "cand_..."]}'

When it settles, the results give each candidate its pass rate with a 95% confidence interval, and its cost per task when the bench can measure it. See Read the result.

Troubleshooting