start an eval run.md
Start an eval run
Queues a durable eval job and returns immediately with a polling ID. Poll GET /v2/eval-runs/{run_id} until its status reaches a terminal value; once finished it also returns the results and analysis. Follow a run live with the monitor operation.
Send one of these flat request forms:
- Saved eval: provide
eval_id; optionally replace its prompts, persona, agents, CLIs, MCPs, setup commands, cleanup commands, or runs for this launch. A saved configuration can still provide repository, environment, skills, and legacy tool selections. - Direct create-and-run: provide
prompts,judges, andagent. You may also provide a persona, tags, repository, write-only environment variables, saved or inline CLIs and MCP servers, skills, setup commands, and cleanup commands.
The run fans out across prompts × agents × runs. Persona is shared metadata and does not multiply attempts. No separate mode field is required. If the organization lacks the estimated credits, the request fails with 402 before dispatching work.
A saved or inline environment containing an empty or placeholder credential is rejected with 400 before any evaluation is dispatched.
OpenAPI
openapi: 3.1.0
info:
title: Lightsage Public API
description: Public API for Lightsage CLI and external integrations.
version: 1.0.0
servers:
- url: https://api.lightsage.com
description: Production
security: []
tags:
- name: Status
description: Check API availability and validate the calling API key.
- name: Visibility
description: >-
Measure visibility, citations, competitors, and prompt performance across
answer platforms.
- name: Prompts
description: Manage tracked prompts and topics, and inspect their stored runs.
- name: Opportunities
description: Review ranked opportunities computed from prompt visibility results.
- name: Tasks
description: Manage the persisted action queue and task lifecycle.
- name: Site Audits
description: Review website audit results and page-level findings.
- name: Agent Analytics
description: >-
See which coding agents, AI bots, and people visit your tracked sites,
which pages they read, which pages return errors, and where the visits
come from.
- name: Evals
description: Define eval tasks, judges, subjects, and saved configurations.
- name: Eval Templates
description: >-
Select reusable judges, targets, schedules, metadata, and runtime settings
with `POST /v2/eval-templates`, then copy only those settings into
prompt-specific evals with `POST /v2/evals/batch`. Templates never contain
prompts and remain independent from evals created from them.
- name: Eval Runs
description: Start eval runs and inspect their lifecycle state.
- name: Eval Results
description: Review completed eval outcomes and diagnostic details.
- name: Agent Preference
description: >-
Compare your company head-to-head with a tracked competitor. Create a
comparison with `POST /v2/agent-preference/evals`, run it with `POST
/v2/eval-runs`, then read which company agents recommended.
- name: Agents
description: List runnable coding-agent harness and model combinations.
- name: Repositories
description: Manage reusable Git repository workspaces.
- name: Environments
description: List reusable runtime environments without exposing secret values.
- name: MCP Servers
description: List reusable MCP server integrations without exposing credentials.
- name: CLIs
description: List reusable command-line interfaces without exposing setup commands.
- name: Skills
description: List reusable skills available to workflows and evals.
paths:
/v2/eval-runs:
post:
tags:
- Eval Runs
summary: Start an eval run
description: >-
Queues a durable eval job and returns immediately with a polling ID.
Poll `GET /v2/eval-runs/{run_id}` until its status reaches a terminal
value; once finished it also returns the results and analysis. Follow a
run live with the monitor operation.
Send one of these flat request forms:
- **Saved eval:** provide `eval_id`; optionally replace its prompts,
persona, agents, CLIs, MCPs, setup commands, cleanup commands, or runs
for this launch. A saved configuration can still provide repository,
environment, skills, and legacy tool selections.
- **Direct create-and-run:** provide `prompts`, `judges`, and `agent`.
You may also provide a persona, tags, repository, write-only environment
variables, saved or inline CLIs and MCP servers, skills, setup commands,
and cleanup commands.
A saved or inline environment containing an empty or placeholder
credential is rejected with `400` before any evaluation is dispatched.
operationId: v2_eval_runs_create
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/CreateV2EvalRunRequest'
examples:
saved_eval:
summary: Run a self-contained saved eval
description: >-
Provide `eval_id` and any environment values required by its
saved MCP definitions. A configuration remains optional.
value:
eval_id: eval_123
env:
LIGHTSAGE_API_KEY: secret-value
direct_eval:
summary: Create and run an eval directly
description: >-
Omit `eval_id` and provide `prompts`, `judges`, and one or
more values in `agent`. The created eval ID is returned with
the run.
value:
agent:
- codex:gpt-5-4
- claude-code:claude-opus-4-8
prompts:
- Complete the task with the CLI.
- Complete the task with the MCP server.
judges:
- The requested workflow succeeds.
- The generated project passes its tests.
persona: Platform engineer
runs: 2
tags:
- sdk
- smoke-test
repository: nextjs-starter
clis:
- brew install lightsagehq/tools/lightsage
- npx -y firecrawl-cli@latest init --all
setup_commands:
- npm install
- npm run db:migrate
cleanup_commands:
- npm run db:cleanup
mcps:
- name: lightsage
type: http
url: https://mcp.lightsage.com/mcp
headers:
X-Lightsage-Api-Key: ${LIGHTSAGE_API_KEY}
env:
LIGHTSAGE_API_KEY: secret-value
skills:
- resend/resend-skills
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/V2EvalRun'
'400':
description: >-
The saved/direct mode fields, referenced resource, or environment
credentials are invalid.
content:
application/json:
schema:
type: object
properties:
detail:
type: string
required:
- detail
examples:
invalid_environment_credentials:
summary: Environment credentials are missing
value:
detail: >-
Environment profile 'Production' contains empty or
placeholder values for: ACME_API_KEY. Update those values
before running the evaluation.
'402':
description: >-
The organization does not currently have enough credits for the
requested run.
content:
application/json:
schema:
type: object
properties:
detail:
type: object
properties:
error:
type: string
example: Insufficient credits
required:
type: number
description: Usage-based credits the requested run needs.
balance:
type: number
description: The organization's current credit balance.
required:
- error
- required
- balance
required:
- detail
'403':
description: >-
The requested coding agent is unavailable on the organization's
current plan.
content:
application/json:
schema:
type: object
properties:
detail:
type: string
required:
- detail
example:
detail: 'Your current plan does not include: cursor'
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- LightsageApiKeyAuth: []
components:
schemas:
CreateV2EvalRunRequest:
properties:
eval_id:
type: string
minLength: 1
title: Eval Id
description: >-
Saved-run form only. Existing eval definition to run. Omit this
field to use the direct create-and-run form.
configuration_id:
type: string
minLength: 1
title: Configuration Id
description: >-
Saved-run form only. Optional saved repository, environment, skills,
and legacy tool configuration.
name:
type: string
maxLength: 160
minLength: 1
title: Name
description: >-
Direct create-and-run form only. Display name for the durable eval.
Defaults to the first prompt.
prompts:
anyOf:
- items:
type: string
type: array
maxItems: 20
minItems: 1
- type: 'null'
title: Prompts
description: >-
Task prompts. Required in direct mode; in saved mode, replaces saved
prompts for this launch. CLI clients expose repeatable `--prompt`.
judges:
anyOf:
- items:
type: string
type: array
maxItems: 20
minItems: 1
- type: 'null'
title: Judges
description: >-
Success criteria. Required in direct mode; in saved mode, replaces
saved judges for this launch. CLI clients expose repeatable
`--judge`.
agent:
anyOf:
- items:
type: string
type: array
maxItems: 10
minItems: 1
- type: 'null'
title: Agent
description: >-
Runnable agents from `GET /v2/agents`. Required unless the saved
eval already has agents. CLI clients expose repeatable `--agent`.
persona:
type: string
maxLength: 160
title: Persona
description: >-
Optional perspective for every attempt in this launch. It does not
modify prompts or add a fan-out dimension.
clis:
anyOf:
- items:
type: string
type: array
maxItems: 20
- type: 'null'
title: Clis
description: >-
Saved CLI IDs or shell installation commands. UUID values resolve
saved CLIs from `GET /v2/clis`; every other value is run as an
installation command. In saved mode, an explicit empty array
disables saved CLIs for this launch. CLI clients expose repeatable
`--cli`.
mcps:
anyOf:
- items:
anyOf:
- $ref: '#/components/schemas/V2SavedMcpReference'
- $ref: '#/components/schemas/V2InlineMcpDefinition'
type: array
maxItems: 20
- type: 'null'
title: Mcps
description: >-
Saved MCP references or inline portable definitions. In saved mode,
an explicit empty array disables saved MCPs for this launch.
setup_commands:
anyOf:
- items:
type: string
type: array
maxItems: 20
- type: 'null'
title: Setup Commands
description: >-
Shell commands run sequentially in the prepared repository before
the agent starts. In saved mode, an explicit empty array disables
saved setup commands for this launch. CLI clients expose repeatable
`--setup-command`.
cleanup_commands:
anyOf:
- items:
type: string
type: array
maxItems: 20
- type: 'null'
title: Cleanup Commands
description: >-
Best-effort shell commands run sequentially after agent execution
and result capture, before sandbox deletion. In saved mode, an
explicit empty array disables saved cleanup commands for this
launch. CLI clients expose repeatable `--cleanup-command`.
runs:
type: integer
maximum: 20
minimum: 1
title: Runs
description: >-
Independent attempts for every prompt-agent pair. Defaults to the
saved eval value or 1 for a direct run.
tags:
items:
type: string
type: array
maxItems: 20
title: Tags
description: >-
Direct create-and-run form only. Optional labels saved on the
created eval. CLI clients may expose this string array as a
repeatable `--tag` flag.
repository:
type: string
minLength: 1
title: Repository
description: >-
Direct create-and-run form only. A saved repository ID, built-in
alias such as `nextjs-starter`, or a public GitHub HTTPS URL.
env:
additionalProperties:
type: string
type: object
maxProperties: 100
title: Env
description: >-
Write-only environment variables for this launch. Values are stored
only in the queued job's private configuration.
writeOnly: true
skills:
items:
type: string
type: array
maxItems: 20
title: Skills
description: >-
Additional public skill packages to install before the agent starts
in either request form, using GitHub `owner/repository` references
such as `resend/resend-skills`. CLI clients may expose this string
array as a repeatable `--skill` flag.
skill_ids:
items:
type: string
type: array
maxItems: 20
title: Skill Ids
description: >-
Direct create-and-run form only. Reusable skill IDs from `GET
/v2/skills` to install before the agent starts.
search_grounding:
anyOf:
- $ref: '#/components/schemas/SearchGroundingMode'
- type: 'null'
description: >-
Web-retrieval policy for this launch. Overrides the selected
configuration or saved eval when provided; direct runs default to
`default`.
additionalProperties: false
type: object
title: CreateV2EvalRunRequest
description: >-
Start an eval run using one of two flat request forms. Saved runs
require `eval_id` and use saved simulation defaults unless the request
replaces them. Direct create-and-run requests require `prompts`,
`judges`, and `agent`. No separate mode field is required.
examples:
- env:
LIGHTSAGE_API_KEY: secret-value
eval_id: eval_123
- agent:
- codex:gpt-5-4
- claude-code:claude-opus-4-8
judges:
- The requested workflow succeeds.
- The generated project passes its tests.
name: SDK simulation
persona: Platform engineer
prompts:
- Complete the task with the CLI.
- Complete the task with the MCP server.
repository: nextjs-starter
runs: 2
skills:
- resend/resend-skills
tags:
- sdk
- smoke-test
V2EvalRun:
properties:
id:
type: string
title: Id
description: Run ID to use when polling eval-run status.
job_id:
type: string
title: Job Id
description: Alias of `id` for the durable parent job.
eval_id:
anyOf:
- type: string
- type: 'null'
title: Eval Id
description: Saved eval definition ID, including one created by a direct run.
eval_name:
anyOf:
- type: string
- type: 'null'
title: Eval Name
description: Display name captured for the eval when the run was created.
configuration_id:
anyOf:
- type: string
- type: 'null'
title: Configuration Id
persona:
anyOf:
- type: string
- type: 'null'
title: Persona
description: Perspective applied to every attempt in this simulation.
agent:
items:
type: string
type: array
title: Agent
description: Runnable agents selected for this simulation.
prompt_count:
type: integer
minimum: 1
title: Prompt Count
description: Number of prompts included in the simulation.
default: 1
runs:
type: integer
minimum: 1
title: Runs
description: Independent attempts requested for every prompt-agent pair.
default: 1
attempt_count:
type: integer
minimum: 1
title: Attempt Count
description: Total concrete attempts after all fan-out dimensions.
default: 1
status:
type: string
title: Status
description: >-
Terminal values: `completed`, `failed`, and `cancelled`. Recoverable
interruptions are reported as `running` with stage `retrying`.
stage:
anyOf:
- type: string
- type: 'null'
title: Stage
dispatch_id:
anyOf:
- type: string
- type: 'null'
title: Dispatch Id
progress:
$ref: '#/components/schemas/PublicEvalRunProgress'
estimated_cost:
anyOf:
- type: number
- type: 'null'
title: Estimated Cost
error:
anyOf:
- type: string
- type: 'null'
title: Error
created_at:
anyOf:
- type: string
- type: 'null'
title: Created At
started_at:
anyOf:
- type: string
- type: 'null'
title: Started At
completed_at:
anyOf:
- type: string
- type: 'null'
title: Completed At
type: object
required:
- id
- job_id
- status
title: V2EvalRun
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
V2SavedMcpReference:
properties:
id:
type: string
minLength: 1
title: Id
description: Saved MCP server ID from `GET /v2/mcp-servers`.
additionalProperties: false
type: object
required:
- id
title: V2SavedMcpReference
V2InlineMcpDefinition:
properties:
name:
type: string
maxLength: 100
minLength: 1
title: Name
description: Stable server name used in the selected agent's MCP config.
type:
type: string
enum:
- http
- sse
- stdio
title: Type
url:
anyOf:
- type: string
- type: 'null'
title: Url
description: HTTPS endpoint for an `http` or `sse` MCP server.
headers:
additionalProperties:
type: string
type: object
maxProperties: 50
title: Headers
description: >-
HTTP headers. Reference run secrets with `${VARIABLE_NAME}` instead
of embedding credentials.
command:
anyOf:
- type: string
- type: 'null'
title: Command
description: Executable command for a `stdio` MCP server.
args:
items:
type: string
type: array
maxItems: 100
title: Args
env:
additionalProperties:
type: string
type: object
maxProperties: 100
title: Env
description: >-
Environment mapping for a `stdio` MCP server. Values may reference
the run environment's variables with `${VARIABLE_NAME}`; secrets
(names ending in KEY, TOKEN, SECRET or PASSWORD) must.
additionalProperties: false
type: object
required:
- name
- type
title: V2InlineMcpDefinition
SearchGroundingMode:
type: string
enum:
- disabled
- enabled
- default
title: SearchGroundingMode
description: Requested web-retrieval behavior for an eval run.
PublicEvalRunProgress:
properties:
total:
type: integer
title: Total
default: 0
queued:
type: integer
title: Queued
default: 0
running:
type: integer
title: Running
default: 0
completed:
type: integer
title: Completed
default: 0
failed:
type: integer
title: Failed
default: 0
cancelled:
type: integer
title: Cancelled
default: 0
retrying:
type: integer
title: Retrying
description: >-
Recoverable child jobs waiting to be retried. These remain active
and are excluded from `percent`.
default: 0
interrupted:
type: integer
title: Interrupted
description: Deprecated compatibility alias for `retrying`; use `retrying`.
default: 0
deprecated: true
percent:
type: number
title: Percent
description: >-
The percentage of child jobs in a final state (`completed`,
`failed`, or `cancelled`). It can reach `100` while the parent job
is still summarizing, so use `status` to detect completion.
default: 0
type: object
title: PublicEvalRunProgress
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
securitySchemes:
LightsageApiKeyAuth:
type: apiKey
in: header
name: X-Lightsage-Api-Key