create evals from a template.md
Create evals from a template
Step 2 of the reusable template workflow. Provide the id and version returned by POST /v2/eval-templates as template_id and expected_template_version, then provide only prompt-specific fields in evals. The template supplies only the settings named in its included_components; omitted settings keep the values supplied by the creation workflow or its defaults. Included values are copied at creation time, so later template changes do not alter the created evals. Competitor selections create one additional eval per selected competitor for every input item, and the response groups each base eval with those variants. Brand-scoped runtime resources are validated against the organization's primary brand before any eval is created.
OpenAPI
openapi: 3.1.0
info:
title: Lightsage Public API
description: Public API for Lightsage CLI and external integrations.
version: 1.0.0
servers:
- url: https://api.lightsage.com
description: Production
security: []
tags:
- name: Status
description: Check API availability and validate the calling API key.
- name: Visibility
description: >-
Measure visibility, citations, competitors, and prompt performance across
answer platforms.
- name: Prompts
description: Manage tracked prompts and topics, and inspect their stored runs.
- name: Opportunities
description: Review ranked opportunities computed from prompt visibility results.
- name: Tasks
description: Manage the persisted action queue and task lifecycle.
- name: Site Audits
description: Review website audit results and page-level findings.
- name: Agent Analytics
description: >-
See which coding agents, AI bots, and people visit your tracked sites,
which pages they read, which pages return errors, and where the visits
come from.
- name: Evals
description: Define eval tasks, judges, subjects, and saved configurations.
- name: Eval Templates
description: >-
Select reusable judges, targets, schedules, metadata, and runtime settings
with `POST /v2/eval-templates`, then copy only those settings into
prompt-specific evals with `POST /v2/evals/batch`. Templates never contain
prompts and remain independent from evals created from them.
- name: Eval Runs
description: Start eval runs and inspect their lifecycle state.
- name: Eval Results
description: Review completed eval outcomes and diagnostic details.
- name: Agent Preference
description: >-
Compare your company head-to-head with a tracked competitor. Create a
comparison with `POST /v2/agent-preference/evals`, run it with `POST
/v2/eval-runs`, then read which company agents recommended.
- name: Agents
description: List runnable coding-agent harness and model combinations.
- name: Repositories
description: Manage reusable Git repository workspaces.
- name: Environments
description: List reusable runtime environments without exposing secret values.
- name: MCP Servers
description: List reusable MCP server integrations without exposing credentials.
- name: CLIs
description: List reusable command-line interfaces without exposing setup commands.
- name: Skills
description: List reusable skills available to workflows and evals.
paths:
/v2/evals/batch:
post:
tags:
- Evals
summary: Create evals from a template
description: >-
Step 2 of the reusable template workflow. Provide the `id` and `version`
returned by `POST /v2/eval-templates` as `template_id` and
`expected_template_version`, then provide only prompt-specific fields in
`evals`. The template supplies only the settings named in its
`included_components`; omitted settings keep the values supplied by the
creation workflow or its defaults. Included values are copied at
creation time, so later template changes do not alter the created evals.
Competitor selections create one additional eval per selected competitor
for every input item, and the response groups each base eval with those
variants. Brand-scoped runtime resources are validated against the
organization's primary brand before any eval is created.
operationId: v2_evals_batch_create
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/CreateEvalTemplateBatchRequest'
required: true
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/EvalTemplateBatchResponse'
'400':
description: >-
The template contains invalid judges, targets, scheduling, metadata,
or organization resources.
content:
application/json:
schema:
type: object
properties:
detail:
type: string
required:
- detail
example:
detail: Unknown or inactive competitor ID
'404':
description: The Eval template does not exist in the organization.
content:
application/json:
schema:
type: object
properties:
detail:
type: string
required:
- detail
example:
detail: Eval template not found
'409':
description: The supplied template version is stale.
content:
application/json:
schema:
type: object
properties:
detail:
type: string
required:
- detail
example:
detail: >-
The Eval template changed after it was loaded. Refresh and try
again.
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- LightsageApiKeyAuth: []
components:
schemas:
CreateEvalTemplateBatchRequest:
properties:
template_id:
type: string
minLength: 1
title: Template Id
description: Template `id` returned by `POST /v2/eval-templates`.
expected_template_version:
type: integer
minimum: 1
title: Expected Template Version
description: >-
Template `version` returned when the template was read or created.
When supplied, creation fails if the template changed before reuse.
evals:
items:
$ref: '#/components/schemas/EvalTemplateBatchItem'
type: array
maxItems: 100
minItems: 1
title: Evals
description: >-
Prompt-specific evals to create from the referenced template.
Template-owned fields are intentionally rejected here to avoid
ambiguous overrides.
additionalProperties: false
type: object
required:
- template_id
- evals
title: CreateEvalTemplateBatchRequest
description: Reuse one template across up to 100 prompt-specific eval definitions.
examples:
- evals:
- name: BTC price fetcher
prompts:
- >-
Write a Python script that prints the current price of BTC
using a free API without a key.
type: workflow
- name: BTC alert
prompts:
- Build a free BTC price alert without an API key.
type: workflow
expected_template_version: 1
template_id: 58f3604e-d58e-4eb8-b1fe-3478be875c36
EvalTemplateBatchResponse:
properties:
template_id:
type: string
title: Template Id
template_version:
type: integer
title: Template Version
created_count:
type: integer
title: Created Count
base_count:
type: integer
title: Base Count
variant_count:
type: integer
title: Variant Count
results:
items:
$ref: '#/components/schemas/EvalTemplateBatchResult'
type: array
title: Results
type: object
required:
- template_id
- template_version
- created_count
- base_count
- variant_count
title: EvalTemplateBatchResponse
examples:
- base_count: 1
created_count: 2
results:
- base:
eval_id: b7e8fd2f-197f-4114-8206-6083d3079752
id: d3336acd-5637-40d0-a1da-89b142590c4c
name: BTC price fetcher
prompt: Fetch the current BTC price.
input_index: 0
name: BTC price fetcher
variants:
- competitor_id: 3f70aa73-bfbe-4494-934d-72a60989115d
competitor_name: Competitor
eval_id: c08e3af8-9bed-4544-81d1-d5c9f3c227cb
id: 8ebcbe4d-c35b-4bce-a1a7-5aff1d02a0fb
name: BTC price fetcher
prompt: Fetch the current BTC price.
template_id: 58f3604e-d58e-4eb8-b1fe-3478be875c36
template_version: 1
variant_count: 1
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
EvalTemplateBatchItem:
properties:
name:
type: string
maxLength: 160
minLength: 1
title: Name
description: Name of the prompt-specific eval to create.
prompts:
items:
type: string
type: array
maxItems: 20
minItems: 1
title: Prompts
description: >-
Task prompts owned by this eval. Judges, targets, scheduling, and
runtime resources come from the template.
type:
type: string
enum:
- workflow
- mcp
- cli
- skill
title: Type
description: Eval subject type used for organization and reporting.
default: workflow
subject_id:
anyOf:
- type: string
- type: 'null'
title: Subject Id
description: Optional subject identifier associated with this eval.
additionalProperties: false
type: object
required:
- name
- prompts
title: EvalTemplateBatchItem
description: Prompt-specific fields combined with the referenced template.
EvalTemplateBatchResult:
properties:
input_index:
type: integer
title: Input Index
name:
type: string
title: Name
base:
$ref: '#/components/schemas/EvalTemplateBatchCreatedEval'
variants:
items:
$ref: '#/components/schemas/EvalTemplateBatchCreatedEval'
type: array
title: Variants
type: object
required:
- input_index
- name
- base
title: EvalTemplateBatchResult
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
EvalTemplateBatchCreatedEval:
properties:
id:
type: string
title: Id
eval_id:
anyOf:
- type: string
- type: 'null'
title: Eval Id
name:
type: string
title: Name
prompt:
type: string
title: Prompt
competitor_id:
anyOf:
- type: string
- type: 'null'
title: Competitor Id
competitor_name:
anyOf:
- type: string
- type: 'null'
title: Competitor Name
type: object
required:
- id
- name
- prompt
title: EvalTemplateBatchCreatedEval
securitySchemes:
LightsageApiKeyAuth:
type: apiKey
in: header
name: X-Lightsage-Api-Key
This documentation is built and hosted on Mintlify, a developer documentation platform.