## Eval commands

| Command | Description |
| --- | --- |
| `lightsage evals list` | List evals. |
| `lightsage evals create` | Create an eval. |
| `lightsage evals retrieve --eval-id <eval_id>` | Retrieve one eval. |
| `lightsage evals update --eval-id <eval_id>` | Update one eval. |
| `lightsage evals delete --eval-id <eval_id>` | Delete one eval. |

## Create an eval

```
lightsage evals create \
  --name "Lightsage interface simulation" \
  --prompt "Use the Lightsage CLI to create an eval." \
  --prompt "Use the Lightsage MCP server to retrieve an eval." \
  --judge "The requested workflow completes successfully." \
  --persona "Platform engineer" \
  --agent codex:gpt-5-4 \
  --agent claude-code:claude-opus-4-8 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp @lightsage-mcp.json \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --runs 2 \
  --tag smoke-test
```

Repeat `--prompt`, `--judge`, `--agent`, `--cli`, `--mcp`, `--setup-command`, `--cleanup-command`, and `--tag` once per value. `--persona` is one shared value. The total attempt count is `prompts × agents × runs` and cannot exceed 100. A `--cli` value can be a saved CLI UUID or an installation command. An `--mcp` value can be a saved MCP UUID, an inline JSON object, or `@path/to/file.json`. Creating an eval does not accept or store secret environment values.Setup commands run sequentially after the repository and declared tools are prepared but before the agent starts. Cleanup commands run best-effort after result capture and before sandbox deletion.

## List and retrieve evals

```
lightsage evals list --output-format json
lightsage evals retrieve --eval-id eval_123 --output-format json
```

Responses use the same public fields: `prompts`, `judges`, `persona`, `agent`, `clis`, `mcps`, `setup_commands`, `cleanup_commands`, `runs`, and `tags`.

## Update an eval

```
lightsage evals update \
  --eval-id eval_123 \
  --body '{
    "prompts": ["Run the narrower scenario."],
    "agent": ["codex:gpt-5-4"],
    "setup_commands": ["npm install"],
    "cleanup_commands": ["npm run db:cleanup"],
    "runs": 3
  }'
```

Omitted fields remain unchanged. Empty `agent`, `clis`, `mcps`, `setup_commands`, `cleanup_commands`, or `tags` arrays clear those saved values. Send `persona: null` in the request body to clear the saved persona.

## Delete an eval

```
lightsage evals delete --eval-id eval_123
```

Deleting an eval also deletes its saved configurations.

## Saved configurations

Configurations remain available for reusable repositories, environment profiles, skills, and legacy tool selections:

```
lightsage evals configurations create \
  --eval-id eval_123 \
  --name "Configured runtime" \
  --repository-id repo_123 \
  --environment-id env_123 \
  --skill-ids skill_123 \
  --skills resend/resend-skills
```

The configuration is optional when the saved eval already contains everything the run needs.