running evals.md

Running evals

Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.

Use lightsage evals run to create and start an eval in one command. One request can test multiple prompts across multiple agents and repeat every pair.

NOTE
The number of attempts is prompts × agents × runs. One request can create at most 100 attempts.

Set up the CLI

Authenticate once, then list the reusable resources available to your organization:

themes={null}
lightsage auth login
lightsage agents list
lightsage repositories list
lightsage clis list
lightsage mcp-servers list

Create and run directly

Use the direct form for a one-off eval:

themes={null}
lightsage evals run \
  --name "Lightsage interface simulation" \
  --prompt "Use the Lightsage CLI to create an eval." \
  --prompt "Use the Lightsage MCP server to retrieve an eval." \
  --judge "The requested workflow completes successfully." \
  --persona "Platform engineer" \
  --agent codex:gpt-5-4 \
  --agent claude-code:claude-opus-4-8 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp '{
    "name": "lightsage",
    "type": "http",
    "url": "https://mcp.lightsage.com/mcp",
    "headers": {
      "X-Lightsage-Api-Key": "${LIGHTSAGE_API_KEY}"
    }
  }' \
  --setup-command "npm install" \
  --setup-command "npm run db:migrate" \
  --cleanup-command "npm run db:cleanup" \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
  --runs 2 \
  --output-format json

This command creates eight attempts: two prompts × two agents × two runs. The persona applies to every attempt without adding another fan-out dimension.

The response contains the persisted eval_id and the parent run id. Keep the run ID to monitor the full matrix.

Attach multiple CLIs and MCP servers

Repeat --cli and --mcp in request order. Each value can reference a saved resource or define one inline:

themes={null}
lightsage evals run \
  --prompt "Exercise every available Lightsage interface." \
  --judge "The workflow completes successfully." \
  --agent codex:gpt-5-4 \
  --cli 230f0c71-ed91-4a4a-9932-9cef267411bc \
  --cli "npm install --global @example/tools" \
  --mcp 550e8400-e29b-41d4-a716-446655440000 \
  --mcp @lightsage-mcp.json \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
  --runs 3

For --cli, a UUID references a saved CLI. Any other value is an installation command.

For --mcp, pass a saved UUID, one inline JSON object, or @path/to/file.json. An MCP file can contain either the canonical object with a name field or a one-server MCP config map:

themes={null}
{
  "lightsage": {
    "type": "http",
    "url": "https://mcp.lightsage.com/mcp",
    "headers": {
      "X-Lightsage-Api-Key": "${LIGHTSAGE_API_KEY}"
    }
  }
}

The ${LIGHTSAGE_API_KEY} reference is resolved from repeated --env KEY=VALUE fields when the sandbox starts. Lightsage does not return or add the resolved value to public run metadata.

Saved MCP IDs and inline MCP definitions use the same runtime path after resolution. Lightsage translates the portable MCP definition into the native configuration for Claude Code, Codex, Cursor, GitHub Copilot, Antigravity, Gemini CLI, OpenClaw, OpenCode, Hermes, Pi, Kilo, and Grok.

Use repeatable --skill owner/repository values to install public agent skills before the selected harness starts. Skill installation and both saved and inline CLI installation commands run during the same pre-execution setup.

Run setup and cleanup commands

Repeat --setup-command to run repository-specific preparation after CLI and skill installation but before the agent starts. Setup commands run in order in the prepared repository. A failed setup command stops the attempt before agent execution.

Repeat --cleanup-command for best-effort teardown after the agent result is captured and before the sandbox is deleted. Cleanup runs for successful and failed agent attempts. If one cleanup command fails, Lightsage records the failure and continues with the remaining cleanup commands without replacing the attempt's primary result.

Prompt an agent to run the eval

Use this prompt when you delegate the workflow to a coding agent:

themes={null}
Use the Lightsage CLI to directly create and run a one-off eval with
`lightsage evals run`.

Create a simple eval about crawling a public webpage about fish. Use
`codex:gpt-5-4`, include a clear success criterion, and run it once. Return the
created eval ID and run ID. Monitor the run until it reaches a terminal state,
then retrieve its analysis.

Create a reusable eval

Save the simulation matrix when you expect to run it again:

themes={null}
lightsage evals create \
  --name "SDK installation" \
  --prompt "Install the SDK and send a successful request." \
  --judge "The SDK is installed and the request succeeds." \
  --persona "Application developer" \
  --agent codex:gpt-5-4 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp @lightsage-mcp.json \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --runs 2 \
  --tag onboarding

Saved evals can contain prompts, judges, agents, persona, CLIs, MCP servers, setup commands, cleanup commands, repeat count, and tags. They do not store inline secret values.

Create a saved configuration only when the eval also needs a reusable repository, environment profile, skill set, or legacy resource selection:

themes={null}
lightsage evals configurations create \
  --eval-id eval_123 \
  --name "Configured runtime" \
  --repository-id repo_123 \
  --environment-id env_123 \
  --skill-ids skill_123 \
  --skills resend/resend-skills

Run the saved defaults:

themes={null}
lightsage evals run \
  --eval-id eval_123 \
  --configuration-id config_123 \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY"

The configuration is optional. You can also override the saved prompts, persona, agents, CLIs, MCP servers, setup commands, cleanup commands, or repeat count for one launch:

themes={null}
lightsage evals run \
  --eval-id eval_123 \
  --prompt "Run this narrower scenario." \
  --agent codex:gpt-5-4 \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --runs 5

In the public API, send an explicit empty setup_commands, cleanup_commands, clis, or mcps array to disable the saved collection for one launch.

Monitor progress

Monitor all active eval runs:

themes={null}
lightsage evals monitor

Monitor one run or retrieve a finite status response:

themes={null}
lightsage evals monitor --run-id run_123
lightsage eval-runs retrieve --run-id run_123 --output-format json

Inspect analysis

After the run reaches a terminal state, retrieve its persisted judge analysis:

themes={null}
lightsage evals analysis --run-id run_123 --output-format json

Each attempt includes its prompt, prompt index, agent, and repeat index so you can compare every matrix cell.