create an eval template.md
Create an Eval template
Step 1 of the reusable template workflow. Creates an organization-owned Eval template containing only the configuration components selected in included_components. Do not include task prompts in the template. The response returns the template id and version; pass them as template_id and expected_template_version to POST /v2/evals/batch with each eval's name and prompts. Applying a template copies its values into the new evals without creating a live link.
OpenAPI
openapi: 3.1.0
info:
title: Lightsage Public API
description: Public API for Lightsage CLI and external integrations.
version: 1.0.0
servers:
- url: https://api.lightsage.com
description: Production
security: []
tags:
- name: Status
description: Check API availability and validate the calling API key.
- name: Visibility
description: >-
Measure visibility, citations, competitors, and prompt performance across
answer platforms.
- name: Prompts
description: Manage tracked prompts and topics, and inspect their stored runs.
- name: Opportunities
description: Review ranked opportunities computed from prompt visibility results.
- name: Tasks
description: Manage the persisted action queue and task lifecycle.
- name: Site Audits
description: Review website audit results and page-level findings.
- name: Agent Analytics
description: >-
See which coding agents, AI bots, and people visit your tracked sites,
which pages they read, which pages return errors, and where the visits
come from.
- name: Evals
description: Define eval tasks, judges, subjects, and saved configurations.
- name: Eval Templates
description: >-
Select reusable judges, targets, schedules, metadata, and runtime settings
with `POST /v2/eval-templates`, then copy only those settings into
prompt-specific evals with `POST /v2/evals/batch`. Templates never contain
prompts and remain independent from evals created from them.
- name: Eval Runs
description: Start eval runs and inspect their lifecycle state.
- name: Eval Results
description: Review completed eval outcomes and diagnostic details.
- name: Agent Preference
description: >-
Compare your company head-to-head with a tracked competitor. Create a
comparison with `POST /v2/agent-preference/evals`, run it with `POST
/v2/eval-runs`, then read which company agents recommended.
- name: Agents
description: List runnable coding-agent harness and model combinations.
- name: Repositories
description: Manage reusable Git repository workspaces.
- name: Environments
description: List reusable runtime environments without exposing secret values.
- name: MCP Servers
description: List reusable MCP server integrations without exposing credentials.
- name: CLIs
description: List reusable command-line interfaces without exposing setup commands.
- name: Skills
description: List reusable skills available to workflows and evals.
paths:
/v2/eval-templates:
post:
tags:
- Eval Templates
summary: Create an Eval template
description: >-
Step 1 of the reusable template workflow. Creates an organization-owned
Eval template containing only the configuration components selected in
`included_components`. Do not include task prompts in the template. The
response returns the template `id` and `version`; pass them as
`template_id` and `expected_template_version` to `POST /v2/evals/batch`
with each eval's name and prompts. Applying a template copies its values
into the new evals without creating a live link.
operationId: v2_eval_templates_create
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/CreateEvalTemplateRequest'
required: true
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/EvalTemplate'
'400':
description: >-
The template contains invalid judges, targets, scheduling, metadata,
or organization resources.
content:
application/json:
schema:
type: object
properties:
detail:
type: string
required:
- detail
example:
detail: Unknown or inactive competitor ID
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- LightsageApiKeyAuth: []
components:
schemas:
CreateEvalTemplateRequest:
properties:
name:
type: string
maxLength: 160
minLength: 1
title: Name
description: Organization-visible name for this reusable template.
description:
type: string
maxLength: 1000
title: Description
description: Optional guidance describing when to use the template.
prompt:
type: string
maxLength: 20000
title: Prompt
description: >-
Prompt filled into the new eval form when the template is applied,
or the scenario for Agent Preference, where `{{competitor_name}}`
names the brand each comparison is against. Requests that create
evals from a template still supply their own prompts.
eval_type:
type: string
enum:
- custom_eval
- agent_preference
title: Eval Type
description: >-
Kind of eval the template creates. A template only applies to evals
of the same type.
default: custom_eval
included_components:
items:
type: string
enum:
- judges
- targets
- persona
- competitors
- tag
- category
- runs
- schedule
- repository
- environment
- search_grounding
- skills
- mcp_servers
- clis
- brand_docs
- github_pr
- runtime_commands
type: array
minItems: 1
title: Included Components
description: >-
Settings copied when the template is applied. Omitted components do
not override the new eval; included components copy their saved
value, including an empty value.
persona:
type: string
maxLength: 500
title: Persona
description: Optional persona copied when `persona` is included.
category_name:
type: string
maxLength: 160
title: Category Name
description: Optional category copied when `category` is included.
tag:
type: string
maxLength: 100
title: Tag
description: Optional tag copied when `tag` is included.
competitor_ids:
items:
type: string
type: array
title: Competitor Ids
description: >-
Competitor IDs that create additional variants when `competitors` is
included.
judges:
items:
$ref: '#/components/schemas/GraderDefinition'
type: array
maxItems: 20
title: Judges
description: >-
Judge definitions copied when `judges` is included. May be empty to
explicitly create the eval without template judges.
target_keys:
items:
type: string
type: array
title: Target Keys
description: >-
Runnable target keys copied when `targets` is included, such as
`agent:codex:gpt-5.6-terra`.
schedule:
$ref: '#/components/schemas/EvalTemplateSchedule'
description: Schedule copied when `schedule` is included.
runs_per_execution:
type: integer
maximum: 20
minimum: 1
title: Runs Per Execution
description: Attempt count copied when `runs` is included.
default: 1
runtime:
$ref: '#/components/schemas/EvalTemplateRuntime'
description: >-
Saved runtime values. Each value is copied only when its matching
component appears in `included_components`.
additionalProperties: false
type: object
required:
- name
- included_components
title: CreateEvalTemplateRequest
description: Create a reusable eval that can be applied to many brands or prompts.
examples:
- description: Reusable policy for crypto recommendation evals.
eval_type: custom_eval
included_components:
- judges
- targets
- schedule
judges:
- blocking: false
criteria: >-
Pass only if CoinMarketCap is recommended as the first and main
choice. Fail otherwise.
name: CoinMarketCap first
type: judge
weight: 1
name: Crypto recommendation
schedule:
enabled: true
frequency: daily
target_keys:
- agent:codex:gpt-5.6-terra
- agent:claude-code:global.anthropic.claude-sonnet-5
EvalTemplate:
properties:
name:
type: string
maxLength: 160
minLength: 1
title: Name
description: Organization-visible name for this reusable template.
description:
anyOf:
- type: string
maxLength: 1000
- type: 'null'
title: Description
description: Optional guidance describing when to use the template.
prompt:
anyOf:
- type: string
maxLength: 20000
- type: 'null'
title: Prompt
description: >-
Prompt filled into the new eval form when the template is applied,
or the scenario for Agent Preference, where `{{competitor_name}}`
names the brand each comparison is against. Requests that create
evals from a template still supply their own prompts.
eval_type:
type: string
enum:
- custom_eval
- agent_preference
title: Eval Type
description: >-
Kind of eval the template creates. A template only applies to evals
of the same type.
default: custom_eval
included_components:
items:
type: string
enum:
- judges
- targets
- persona
- competitors
- tag
- category
- runs
- schedule
- repository
- environment
- search_grounding
- skills
- mcp_servers
- clis
- brand_docs
- github_pr
- runtime_commands
type: array
minItems: 1
title: Included Components
description: >-
Settings copied when the template is applied. Omitted components do
not override the new eval; included components copy their saved
value, including an empty value.
persona:
anyOf:
- type: string
maxLength: 500
- type: 'null'
title: Persona
description: Optional persona copied when `persona` is included.
category_name:
anyOf:
- type: string
maxLength: 160
- type: 'null'
title: Category Name
description: Optional category copied when `category` is included.
tag:
anyOf:
- type: string
maxLength: 100
- type: 'null'
title: Tag
description: Optional tag copied when `tag` is included.
competitor_ids:
items:
type: string
type: array
title: Competitor Ids
description: >-
Competitor IDs that create additional variants when `competitors` is
included.
judges:
items:
$ref: '#/components/schemas/GraderDefinition'
type: array
maxItems: 20
title: Judges
description: >-
Judge definitions copied when `judges` is included. May be empty to
explicitly create the eval without template judges.
target_keys:
items:
type: string
type: array
title: Target Keys
description: >-
Runnable target keys copied when `targets` is included, such as
`agent:codex:gpt-5.6-terra`.
schedule:
$ref: '#/components/schemas/EvalTemplateSchedule'
description: Schedule copied when `schedule` is included.
runs_per_execution:
type: integer
maximum: 20
minimum: 1
title: Runs Per Execution
description: Attempt count copied when `runs` is included.
default: 1
runtime:
$ref: '#/components/schemas/EvalTemplateRuntime'
description: >-
Saved runtime values. Each value is copied only when its matching
component appears in `included_components`.
id:
type: string
title: Id
org_id:
type: string
title: Org Id
version:
type: integer
title: Version
created_by:
anyOf:
- type: string
- type: 'null'
title: Created By
created_at:
anyOf:
- type: string
format: date-time
- type: string
title: Created At
updated_at:
anyOf:
- type: string
format: date-time
- type: string
title: Updated At
deleted_at:
anyOf:
- type: string
format: date-time
- type: string
- type: 'null'
title: Deleted At
additionalProperties: false
type: object
required:
- name
- included_components
- id
- org_id
- version
- created_at
- updated_at
title: EvalTemplate
examples:
- created_at: '2026-09-24T18:00:00Z'
created_by: 0b6d843f-cd4e-4820-8999-5c6dac8ca6f2
description: Reusable policy for crypto recommendation evals.
eval_type: custom_eval
id: 58f3604e-d58e-4eb8-b1fe-3478be875c36
included_components:
- judges
- targets
- schedule
judges:
- blocking: false
criteria: >-
Pass only if CoinMarketCap is recommended as the first and main
choice. Fail otherwise.
name: CoinMarketCap first
type: judge
weight: 1
name: Crypto recommendation
org_id: ab9fe05c-5224-4d9b-a382-a23834ec1947
schedule:
enabled: true
frequency: daily
target_keys:
- agent:codex:gpt-5.6-terra
- agent:claude-code:global.anthropic.claude-sonnet-5
updated_at: '2026-09-24T18:00:00Z'
version: 1
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
GraderDefinition:
properties:
name:
type: string
title: Name
type:
type: string
title: Type
needle:
anyOf:
- type: string
- type: 'null'
title: Needle
needles:
anyOf:
- items:
type: string
type: array
- type: 'null'
title: Needles
pattern:
anyOf:
- type: string
- type: 'null'
title: Pattern
case_sensitive:
type: boolean
title: Case Sensitive
default: false
criteria:
anyOf:
- type: string
- type: 'null'
title: Criteria
metric:
anyOf:
- type: string
enum:
- tool_calls
- time
- cost
- type: 'null'
title: Metric
min_score:
anyOf:
- type: number
maximum: 1
minimum: 0
- type: 'null'
title: Min Score
max_value:
anyOf:
- type: number
exclusiveMinimum: 0
- type: 'null'
title: Max Value
weight:
type: number
title: Weight
default: 1
blocking:
type: boolean
title: Blocking
description: >-
Whether a failed or missing result from this grader fails the entire
eval regardless of the aggregate score.
default: false
type: object
required:
- name
- type
title: GraderDefinition
EvalTemplateSchedule:
properties:
enabled:
type: boolean
title: Enabled
description: Whether each eval created from the template receives a schedule.
default: false
frequency:
type: string
enum:
- daily
- every_other_day
- weekly
title: Frequency
description: Schedule cadence used when `enabled` is true.
default: weekly
additionalProperties: false
type: object
title: EvalTemplateSchedule
description: Schedule copied into every eval created from the template.
EvalTemplateRuntime:
properties:
starting_project_id:
anyOf:
- type: string
- type: 'null'
title: Starting Project Id
description: Optional reusable repository workspace ID.
environment_profile_ids:
items:
type: string
type: array
title: Environment Profile Ids
description: Environment profile IDs copied into each created eval.
include_env_vars:
type: boolean
title: Include Env Vars
description: Whether selected environment profiles are enabled.
default: true
search_grounding:
$ref: '#/components/schemas/SearchGroundingMode'
description: Web-retrieval policy copied into each created eval.
default: default
skill_ids:
items:
type: string
type: array
title: Skill Ids
description: Reusable skill IDs copied into each created eval.
mcp_server_ids:
items:
type: string
type: array
title: Mcp Server Ids
description: Reusable MCP server IDs copied into each created eval.
cli_ids:
items:
type: string
type: array
title: Cli Ids
description: Reusable CLI IDs copied into each created eval.
include_brand_docs:
type: boolean
title: Include Brand Docs
description: Whether organization documentation is included at runtime.
default: false
run_on_github_pr:
type: boolean
title: Run On Github Pr
description: Whether each created eval is configured for GitHub pull requests.
default: false
setup_commands:
items:
type: string
type: array
maxItems: 20
title: Setup Commands
description: Shell commands run before each eval attempt, in order.
cleanup_commands:
items:
type: string
type: array
maxItems: 20
title: Cleanup Commands
description: Shell commands run after each eval attempt, in order.
additionalProperties: false
type: object
title: EvalTemplateRuntime
description: Reusable runtime resources copied into each created eval.
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
SearchGroundingMode:
type: string
enum:
- disabled
- enabled
- default
title: SearchGroundingMode
description: Requested web-retrieval behavior for an eval run.
securitySchemes:
LightsageApiKeyAuth:
type: apiKey
in: header
name: X-Lightsage-Api-Key