Eval runs - Lightsage

Documentation Index

Fetch the complete documentation index at: /docs/llms.txt

Use this file to discover all available pages before exploring further.

lightsage evals run either runs a saved eval or creates and starts one directly. A run fans out across every prompt and agent, then repeats each pair according to --runs.

Commands

Command	Description
`lightsage evals run`	Start a saved or direct eval run.
`lightsage evals monitor`	Stream live progress for active runs.
`lightsage eval-runs list`	List eval runs.
`lightsage eval-runs retrieve --run-id <id>`	Retrieve or poll one run.
`lightsage evals analysis --run-id <id>`	Retrieve persisted judge analysis.

Create and run directly

lightsage evals run \ 
  --prompt "Use the CLI to create an eval." \ 
  --prompt "Use the MCP server to retrieve an eval." \ 
  --judge "The requested workflow succeeds." \ 
  --agent codex:gpt-5-4 \ 
  --agent claude-code:claude-opus-4-8 \ 
  --cli "brew install lightsagehq/tools/lightsage" \ 
  --mcp @lightsage-mcp.json \ 
  --setup-command "npm install" \ 
  --cleanup-command "npm run db:cleanup" \ 
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \ 
  --runs 2

Repeat --prompt, --judge, --agent, --cli, --mcp, --setup-command, --cleanup-command, --env, --skill, and --tag once per value. The response includes the created eval_id and the parent run id. Setup commands run sequentially before agent execution and fail the attempt fast if one exits unsuccessfully. Cleanup commands run best-effort after result capture and before sandbox deletion, including when agent execution fails.

Run a saved eval

Run the saved defaults:

lightsage evals run --eval-id eval_123

Add a configuration when the run needs a saved repository, environment profile, skill set, or legacy tool selection:

lightsage evals run \ 
  --eval-id eval_123 \ 
  --configuration-id config_123

You can replace saved prompts, persona, agents, CLIs, MCP servers, setup commands, cleanup commands, or repeat count for one launch. An explicitly empty clis, mcps, setup_commands, or cleanup_commands array in a JSON body disables the saved collection for that launch.

Monitor and inspect a run

lightsage evals monitor --run-id run_123
lightsage eval-runs retrieve --run-id run_123 --output-format json
lightsage evals analysis --run-id run_123 --output-format json

The parent run reports its agents, persona, prompt count, runs per matrix cell, attempt count, status, and progress. Attempt-level analysis and trace records include the exact prompt, prompt index, agent, and repeat index. Inline environment values are write-only. Lightsage keeps them in private job configuration and does not return them in run, trace, result, or analysis responses.