retrieve an eval run.md
Retrieve an eval run
Returns the run's current status and progress. Once its attempts finish, the response also includes their results and analysis. Poll this until the status reaches a terminal value. Use the trace operation for the raw execution event log, or the monitor operation to follow a run live.
OpenAPI
openapi: 3.1.0
info:
title: Lightsage Public API
description: Public API for Lightsage CLI and external integrations.
version: 1.0.0
servers:
- url: https://api.lightsage.com
description: Production
security: []
tags:
- name: Status
description: Check API availability and validate the calling API key.
- name: Visibility
description: >-
Measure visibility, citations, competitors, and prompt performance across
answer platforms.
- name: Prompts
description: Manage tracked prompts and topics, and inspect their stored runs.
- name: Opportunities
description: Review ranked opportunities computed from prompt visibility results.
- name: Tasks
description: Manage the persisted action queue and task lifecycle.
- name: Site Audits
description: Review website audit results and page-level findings.
- name: Agent Analytics
description: >-
See which coding agents, AI bots, and people visit your tracked sites,
which pages they read, which pages return errors, and where the visits
come from.
- name: Evals
description: Define eval tasks, judges, subjects, and saved configurations.
- name: Eval Templates
description: >-
Select reusable judges, targets, schedules, metadata, and runtime settings
with `POST /v2/eval-templates`, then copy only those settings into
prompt-specific evals with `POST /v2/evals/batch`. Templates never contain
prompts and remain independent from evals created from them.
- name: Eval Runs
description: Start eval runs and inspect their lifecycle state.
- name: Eval Results
description: Review completed eval outcomes and diagnostic details.
- name: Agent Preference
description: >-
Compare your company head-to-head with a tracked competitor. Create a
comparison with `POST /v2/agent-preference/evals`, run it with `POST
/v2/eval-runs`, then read which company agents recommended.
- name: Agents
description: List runnable coding-agent harness and model combinations.
- name: Repositories
description: Manage reusable Git repository workspaces.
- name: Environments
description: List reusable runtime environments without exposing secret values.
- name: MCP Servers
description: List reusable MCP server integrations without exposing credentials.
- name: CLIs
description: List reusable command-line interfaces without exposing setup commands.
- name: Skills
description: List reusable skills available to workflows and evals.
paths:
/v2/eval-runs/{run_id}:
get:
tags:
- Eval Runs
summary: Retrieve an eval run
description: >-
Returns the run's current status and progress. Once its attempts finish,
the response also includes their results and analysis. Poll this until
the status reaches a terminal value. Use the trace operation for the raw
execution event log, or the monitor operation to follow a run live.
operationId: v2_eval_runs_retrieve
parameters:
- name: run_id
in: path
required: true
schema:
type: string
title: Run Id
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/V2EvalRunDetail'
'404':
description: >-
Eval run not found. Use the run ID returned when you started the
eval.
content:
application/json:
schema:
type: object
properties:
detail:
type: string
required:
- detail
example:
detail: >-
Eval run not found. Use the run ID returned when you started
the eval.
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- LightsageApiKeyAuth: []
components:
schemas:
V2EvalRunDetail:
properties:
id:
type: string
title: Id
description: Run ID to use when polling eval-run status.
job_id:
type: string
title: Job Id
description: Alias of `id` for the durable parent job.
eval_id:
anyOf:
- type: string
- type: 'null'
title: Eval Id
description: Saved eval definition ID, including one created by a direct run.
eval_name:
anyOf:
- type: string
- type: 'null'
title: Eval Name
description: Display name captured for the eval when the run was created.
configuration_id:
anyOf:
- type: string
- type: 'null'
title: Configuration Id
persona:
anyOf:
- type: string
- type: 'null'
title: Persona
description: Perspective applied to every attempt in this simulation.
agent:
items:
type: string
type: array
title: Agent
description: Runnable agents selected for this simulation.
prompt_count:
type: integer
minimum: 1
title: Prompt Count
description: Number of prompts included in the simulation.
default: 1
runs:
type: integer
minimum: 1
title: Runs
description: Independent attempts requested for every prompt-agent pair.
default: 1
attempt_count:
type: integer
minimum: 1
title: Attempt Count
description: Total concrete attempts after all fan-out dimensions.
default: 1
status:
type: string
title: Status
description: >-
Terminal values: `completed`, `failed`, and `cancelled`. Recoverable
interruptions are reported as `running` with stage `retrying`.
stage:
anyOf:
- type: string
- type: 'null'
title: Stage
dispatch_id:
anyOf:
- type: string
- type: 'null'
title: Dispatch Id
progress:
$ref: '#/components/schemas/PublicEvalRunProgress'
estimated_cost:
anyOf:
- type: number
- type: 'null'
title: Estimated Cost
error:
anyOf:
- type: string
- type: 'null'
title: Error
created_at:
anyOf:
- type: string
- type: 'null'
title: Created At
started_at:
anyOf:
- type: string
- type: 'null'
title: Started At
completed_at:
anyOf:
- type: string
- type: 'null'
title: Completed At
results:
items:
$ref: '#/components/schemas/V2EvalResultSummary'
type: array
title: Results
description: >-
Per-attempt outcomes for this run. Empty while the run is active;
populated as attempts finish.
analysis:
items:
$ref: '#/components/schemas/V2EvalRunAnalysis'
type: array
title: Analysis
description: >-
Per-attempt analysis for this run. Empty until an attempt has been
analyzed.
results_truncated:
type: boolean
title: Results Truncated
description: >-
True when the run produced more results than the inline cap. The
remaining results are not included in this response.
default: false
type: object
required:
- id
- job_id
- status
title: V2EvalRunDetail
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
PublicEvalRunProgress:
properties:
total:
type: integer
title: Total
default: 0
queued:
type: integer
title: Queued
default: 0
running:
type: integer
title: Running
default: 0
completed:
type: integer
title: Completed
default: 0
failed:
type: integer
title: Failed
default: 0
cancelled:
type: integer
title: Cancelled
default: 0
retrying:
type: integer
title: Retrying
description: >-
Recoverable child jobs waiting to be retried. These remain active
and are excluded from `percent`.
default: 0
interrupted:
type: integer
title: Interrupted
description: Deprecated compatibility alias for `retrying`; use `retrying`.
default: 0
deprecated: true
percent:
type: number
title: Percent
description: >-
The percentage of child jobs in a final state (`completed`,
`failed`, or `cancelled`). It can reach `100` while the parent job
is still summarizing, so use `status` to detect completion.
default: 0
type: object
title: PublicEvalRunProgress
V2EvalResultSummary:
properties:
id:
type: string
title: Id
description: Result ID, not an eval job ID.
eval_id:
anyOf:
- type: string
- type: 'null'
title: Eval Id
agent:
anyOf:
- type: string
- type: 'null'
title: Agent
description: Runnable agent combining the harness and model.
prompt:
anyOf:
- type: string
- type: 'null'
title: Prompt
prompt_index:
type: integer
minimum: 0
title: Prompt Index
default: 0
repeat_index:
type: integer
minimum: 0
title: Repeat Index
default: 0
passed:
anyOf:
- type: boolean
- type: 'null'
title: Passed
score:
anyOf:
- type: number
- type: 'null'
title: Score
description: Score from 0 to 1.
failure_reasons:
anyOf:
- items:
type: string
type: array
- type: 'null'
title: Failure Reasons
description: >-
Checks that failed, listed even when the result passed overall.
Deductions from checks that passed are in
`grader_results[].deductions`, not here.
grader_results:
items:
$ref: '#/components/schemas/PublicGraderResult'
type: array
title: Grader Results
description: >-
Per-check verdicts behind `score`, including deductions recorded by
passing checks.
preference_result:
anyOf:
- $ref: '#/components/schemas/V2PreferenceResult'
- type: 'null'
description: >-
Which company the agent recommended. Present only for Agent
Preference evals that produced a usable recommendation.
latency_ms:
anyOf:
- type: integer
- type: 'null'
title: Latency Ms
search_grounding:
anyOf:
- $ref: '#/components/schemas/SearchGroundingMode'
- type: 'null'
web_retrieval_observed:
anyOf:
- type: boolean
- type: 'null'
title: Web Retrieval Observed
search_observed:
anyOf:
- type: boolean
- type: 'null'
title: Search Observed
grounding_qualified:
anyOf:
- type: boolean
- type: 'null'
title: Grounding Qualified
description: >-
Whether the attempt satisfied the requested grounding experiment.
Null means the default mode was used or isolation could not be
proven.
grounding_qualification_reason:
anyOf:
- type: string
- type: 'null'
title: Grounding Qualification Reason
created_at:
anyOf:
- type: string
- type: 'null'
title: Created At
environment:
anyOf:
- $ref: '#/components/schemas/PublicEvalRunEnvironment'
- type: 'null'
description: >-
The environment this attempt used, without secret values. Null when
the attempt had no environment.
type: object
required:
- id
title: V2EvalResultSummary
V2EvalRunAnalysis:
properties:
id:
type: string
title: Id
description: Persisted eval analysis ID.
result_id:
type: string
title: Result Id
description: Concrete eval result ID analyzed for this attempt.
attempt_id:
type: string
title: Attempt Id
description: >-
Child eval-attempt ID used to correlate the analysis with trace
events.
eval_id:
anyOf:
- type: string
- type: 'null'
title: Eval Id
description: Public eval definition ID, not the backing executable eval ID.
agent:
anyOf:
- type: string
- type: 'null'
title: Agent
description: Runnable agent used for this concrete attempt.
prompt:
anyOf:
- type: string
- type: 'null'
title: Prompt
description: Exact task prompt used for this concrete attempt.
prompt_index:
type: integer
minimum: 0
title: Prompt Index
description: Zero-based prompt index within the submitted simulation.
default: 0
variant_key:
type: string
title: Variant Key
description: Runtime variant that produced this result.
default: default
repeat_index:
type: integer
minimum: 0
title: Repeat Index
description: Zero-based repeat index within the submitted run.
default: 0
analysis_version:
anyOf:
- type: string
- type: 'null'
title: Analysis Version
description: Version of the analysis pipeline that produced this payload.
analysis:
additionalProperties: true
type: object
title: Analysis
description: >-
Persisted eval_run_analysis payload, including verdict, outcome,
findings, bottleneck, and analysis sections.
created_at:
anyOf:
- type: string
- type: 'null'
title: Created At
updated_at:
anyOf:
- type: string
- type: 'null'
title: Updated At
type: object
required:
- id
- result_id
- attempt_id
title: V2EvalRunAnalysis
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
PublicGraderResult:
properties:
name:
type: string
title: Name
description: Grader name as configured on the eval.
label:
anyOf:
- type: string
- type: 'null'
title: Label
description: Display name for the check, when the grader sets one.
passed:
anyOf:
- type: boolean
- type: 'null'
title: Passed
description: Whether this check passed.
score:
anyOf:
- type: number
- type: 'null'
title: Score
description: >-
Score from 0 to 1 for checks that score partially, such as judges;
null for pass/fail checks.
deductions:
items:
type: string
type: array
title: Deductions
description: >-
Specific shortcomings the check recorded, even when it passed. A
judge can pass with a score below 1 and still list deductions here;
those checks do not appear in `failure_reasons`.
details:
anyOf:
- type: string
- type: 'null'
title: Details
description: Reasoning or check output explaining the verdict.
criteria_verdicts:
items:
$ref: '#/components/schemas/PublicCriterionVerdict'
type: array
title: Criteria Verdicts
description: >-
Judges only: a verdict for each mandatory requirement in the judge
criteria. Empty for other checks and for runs graded before
per-requirement verdicts were recorded.
type: object
required:
- name
title: PublicGraderResult
V2PreferenceResult:
properties:
outcome:
type: string
enum:
- won
- lost
- tie
title: Outcome
description: >-
`won` when the agent recommended your company, `lost` when it
recommended the competitor, and `tie` when it recommended both
equally or neither.
winner_name:
anyOf:
- type: string
- type: 'null'
title: Winner Name
description: The recommended company. Null on a tie.
competitor_name:
anyOf:
- type: string
- type: 'null'
title: Competitor Name
competitor_domain:
anyOf:
- type: string
- type: 'null'
title: Competitor Domain
type: object
required:
- outcome
title: V2PreferenceResult
description: Which company an agent recommended, from your company's side.
SearchGroundingMode:
type: string
enum:
- disabled
- enabled
- default
title: SearchGroundingMode
description: Requested web-retrieval behavior for an eval run.
PublicEvalRunEnvironment:
properties:
id:
anyOf:
- type: string
- type: 'null'
title: Id
description: Environment ID, as listed by the environments operation.
name:
anyOf:
- type: string
- type: 'null'
title: Name
description: Environment name when the run was queued.
updated_at:
anyOf:
- type: string
- type: 'null'
title: Updated At
description: >-
Environment version the run used: when its name or variables last
changed before the run was queued. Null if the environment had not
changed since versioning was introduced.
queued_at:
anyOf:
- type: string
- type: 'null'
title: Queued At
description: When the run was queued with this environment.
variable_names:
items:
type: string
type: array
title: Variable Names
description: Variable names supplied to the run. Values are never returned.
inline:
type: boolean
title: Inline
description: >-
True when variables were supplied inline with the run request
instead of from a saved environment.
default: false
changed_since_queued:
anyOf:
- type: boolean
- type: 'null'
title: Changed Since Queued
description: >-
True when the saved environment has been edited or deleted since the
run was queued, so a rerun would use different values. Null when
unknown or for inline environments.
type: object
title: PublicEvalRunEnvironment
description: Non-secret summary of the environment an eval run was queued with.
PublicCriterionVerdict:
properties:
criterion:
type: string
title: Criterion
description: One mandatory requirement from the judge criteria.
verdict:
type: string
enum:
- PASS
- FAIL
- NOT_VERIFIED
title: Verdict
description: >-
`PASS` when the evidence shows the requirement was met, `FAIL` when
it shows it was not, and `NOT_VERIFIED` when the evidence cannot
confirm it. Any `FAIL` or `NOT_VERIFIED` fails the run, whatever the
score.
evidence:
anyOf:
- type: string
- type: 'null'
title: Evidence
description: Short evidence the judge gave for the verdict.
type: object
required:
- criterion
- verdict
title: PublicCriterionVerdict
securitySchemes:
LightsageApiKeyAuth:
type: apiKey
in: header
name: X-Lightsage-Api-Key