## Documentation Index

Fetch the complete documentation index at: [/docs/llms.txt](/content/docs/llms.txt)

Use this file to discover all available pages before exploring further.

`lightsage evals run` either runs a saved [eval](/content/docs/cli-reference/evals/index.html) or creates
and starts one directly. A run fans out across every prompt and agent, then
repeats each pair according to `--runs`.

## Commands

| Command                                                    | Description                                  |
|-----------------------------------------------------------|----------------------------------------------|
| `lightsage evals run`                                    | Start a saved or direct eval run.           |
| `lightsage evals monitor`                                 | Stream live progress for active runs.       |
| `lightsage eval-runs list`                                | List eval runs.                             |
| `lightsage eval-runs retrieve --run-id <id>`           | Retrieve or poll one run.                   |
| `lightsage evals analysis --run-id <id>`                 | Retrieve persisted judge analysis.          |

## Create and run directly

```
lightsage evals run \ 
  --prompt "Use the CLI to create an eval." \ 
  --prompt "Use the MCP server to retrieve an eval." \ 
  --judge "The requested workflow succeeds." \ 
  --agent codex:gpt-5-4 \ 
  --agent claude-code:claude-opus-4-8 \ 
  --cli "brew install lightsagehq/tools/lightsage" \ 
  --mcp @lightsage-mcp.json \ 
  --setup-command "npm install" \ 
  --cleanup-command "npm run db:cleanup" \ 
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \ 
  --runs 2
```

Repeat `--prompt`, `--judge`, `--agent`, `--cli`, `--mcp`, `--setup-command`,
`--cleanup-command`, `--env`, `--skill`, and `--tag` once per value. The
response includes the created `eval_id` and the parent run `id`. Setup commands run
sequentially before agent execution and fail the attempt fast if one exits unsuccessfully. Cleanup commands run best-effort after result
capture and before sandbox deletion, including when agent execution fails.

## Run a saved eval

Run the saved defaults:

```
lightsage evals run --eval-id eval_123
```

Add a configuration when the run needs a saved repository, environment
profile, skill set, or legacy tool selection:

```
lightsage evals run \ 
  --eval-id eval_123 \ 
  --configuration-id config_123
```

You can replace saved prompts, persona, agents, CLIs, MCP servers, setup
commands, cleanup commands, or repeat count for one launch. An explicitly empty
`clis`, `mcps`, `setup_commands`, or `cleanup_commands` array in a JSON body
disables the saved collection for that launch.

## Monitor and inspect a run

```
lightsage evals monitor --run-id run_123
lightsage eval-runs retrieve --run-id run_123 --output-format json
lightsage evals analysis --run-id run_123 --output-format json
```

The parent run reports its agents, persona, prompt count, runs per matrix cell,
attempt count, status, and progress. Attempt-level analysis and trace records
include the exact prompt, prompt index, agent, and repeat index. Inline environment values are write-only. Lightsage keeps them in private job
configuration and does not return them in run, trace, result, or analysis
responses.