Running evals - Lightsage
Documentation Index
Fetch the complete documentation index at: /docs/llms.txt
Use this file to discover all available pages before exploring further.
Use lightsage evals run to create and start an eval in one command. One
request can test multiple prompts across multiple agents and repeat every pair.
The number of attempts is prompts × agents × runs. One request can create
at most 100 attempts.
Set up the CLI
Authenticate once, then list the reusable resources available to your organization:
lightsage auth login
lightsage agents list
lightsage repositories list
lightsage clis list
lightsage mcp-servers list
Create and run directly
Use the direct form for a one-off eval:
lightsage evals run \
--name "Lightsage interface simulation" \
--prompt "Use the Lightsage CLI to create an eval." \
--prompt "Use the Lightsage MCP server to retrieve an eval." \
--judge "The requested workflow completes successfully." \
--persona "Platform engineer" \
--agent codex:gpt-5-4 \
--agent claude-code:claude-opus-4-8 \
--cli "brew install lightsagehq/tools/lightsage" \
--mcp '{
"name": "lightsage",
"type": "http",
"url": "https://mcp.lightsage.com/mcp",
"headers": {
"X-Lightsage-Api-Key": "${LIGHTSAGE_API_KEY}"
}
}' \
--setup-command "npm install" \
--setup-command "npm run db:migrate" \
--cleanup-command "npm run db:cleanup" \
--env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
--runs 2 \
--output-format json
This command creates eight attempts: two prompts × two agents × two runs. The
persona applies to every attempt without adding another fan-out dimension.The response contains the persisted eval_id and the parent run id. Keep the
run ID to monitor the full matrix.
Attach multiple CLIs and MCP servers
Repeat --cli and --mcp in request order. Each value can reference a saved
resource or define one inline:
lightsage evals run \
--prompt "Exercise every available Lightsage interface." \
--judge "The workflow completes successfully." \
--agent codex:gpt-5-4 \
--cli 230f0c71-ed91-4a4a-9932-9cef267411bc \
--cli "npm install --global @example/tools" \
--mcp 550e8400-e29b-41d4-a716-446655440000 \
--mcp @lightsage-mcp.json \
--env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
--runs 3
For --cli, a UUID references a saved CLI. Any other value is an installation
command.For --mcp, pass a saved UUID, one inline JSON object, or @path/to/file.json.
An MCP file can contain either the canonical object with a name field or a
one-server MCP config map:
{
"lightsage": {
"type": "http",
"url": "https://mcp.lightsage.com/mcp",
"headers": {
"X-Lightsage-Api-Key": "${LIGHTSAGE_API_KEY}"
}
}
}
The ${LIGHTSAGE_API_KEY} reference is resolved from repeated --env KEY=VALUE
fields when the sandbox starts. Lightsage does not return or add the resolved
value to public run metadata.Saved MCP IDs and inline MCP definitions use the same runtime path after
resolution. Lightsage translates the portable MCP definition into the native
configuration for Claude Code, Codex, Cursor, GitHub Copilot, Antigravity,
Gemini CLI, OpenClaw, OpenCode, Hermes, Pi, Kilo, and Grok.Use repeatable --skill owner/repository values to install public agent skills
before the selected harness starts. Skill installation and both saved and
inline CLI installation commands run during the same pre-execution setup.
Run setup and cleanup commands
Repeat --setup-command to run repository-specific preparation after CLI and
skill installation but before the agent starts. Setup commands run in order in
the prepared repository. A failed setup command stops the attempt before agent
execution.Repeat --cleanup-command for best-effort teardown after the agent result is
captured and before the sandbox is deleted. Cleanup runs for successful and
failed agent attempts. If one cleanup command fails, Lightsage records the
failure and continues with the remaining cleanup commands without replacing
the attempt’s primary result.
Prompt an agent to run the eval
Use this prompt when you delegate the workflow to a coding agent:
Lightsage CLI
Use the Lightsage CLI to directly create and run a one-off eval with
`lightsage evals run`.
Create a simple eval about crawling a public webpage about fish. Use
`codex:gpt-5-4`, include a clear success criterion, and run it once. Return the
created eval ID and run ID. Monitor the run until it reaches a terminal state,
then retrieve its analysis.
Create a reusable eval
Save the simulation matrix when you expect to run it again:
lightsage evals create \
--name "SDK installation" \
--prompt "Install the SDK and send a successful request." \
--judge "The SDK is installed and the request succeeds." \
--persona "Application developer" \
--agent codex:gpt-5-4 \
--cli "brew install lightsagehq/tools/lightsage" \
--mcp @lightsage-mcp.json \
--setup-command "npm install" \
--cleanup-command "npm run db:cleanup" \
--runs 2 \
--tag onboarding
Saved evals can contain prompts, judges, agents, persona, CLIs, MCP servers, setup commands, cleanup commands, repeat count, and tags. They do not store inline secret values.Create a saved configuration only when the eval also needs a reusable repository, environment profile, skill set, or legacy resource selection:
lightsage evals configurations create \
--eval-id eval_123 \
--name "Configured runtime" \
--repository-id repo_123 \
--environment-id env_123 \
--skill-ids skill_123 \
--skills resend/resend-skills
Run the saved defaults:
lightsage evals run \
--eval-id eval_123 \
--configuration-id config_123 \
--env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY"
The configuration is optional. You can also override the saved prompts, persona, agents, CLIs, MCP servers, setup commands, cleanup commands, or repeat count for one launch:
lightsage evals run \
--eval-id eval_123 \
--prompt "Run this narrower scenario." \
--agent codex:gpt-5-4 \
--setup-command "npm install" \
--cleanup-command "npm run db:cleanup" \
--runs 5
In the public API, send an explicit empty setup_commands, cleanup_commands,
clis, or mcps array to disable the saved collection for one launch.
Monitor progress
Monitor all active eval runs:
lightsage evals monitor
Monitor one run or retrieve a finite status response:
lightsage evals monitor --run-id run_123
lightsage eval-runs retrieve --run-id run_123 --output-format json
Inspect analysis
After the run reaches a terminal state, retrieve its persisted judge analysis:
lightsage evals analysis --run-id run_123 --output-format json
Each attempt includes its prompt, prompt index, agent, and repeat index so you can compare every matrix cell.
Related reference
Evals \ \ Create reusable simulation definitions and configurations.
Eval runs \ \ Start runs and monitor their progress.
CLIs \ \ Find saved CLI IDs that can be attached to an eval.
MCP servers \ \ Find saved MCP server IDs that can be attached to an eval.
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Lightsage","colors":{"primary":"#0EA5E9","light":"#38BDF8","dark":"#0284C7"},"logo":{"light":"https://mintcdn.com/sapient-14d4e375/0RQiSxJPQv6MyuZt/images/lightsage-logo-with-text-light.svg?fit=max\\u0026auto=format\\u0026n=0RQiSxJPQv6MyuZt\\u0026q=85\\u0026s=843d0563baa54c778891f942711ab938","dark":"https://mintcdn.com/sapient-14d4e375/0RQiSxJPQv6MyuZt/images/lightsage-logo-with-text-dark.svg?fit=max\\u0026auto=format\\u0026n=0RQiSxJPQv6MyuZt\\u0026q=85\\u0026s=0ec933f6e5153f8392cb4c5f19b7017f"},"favicon":"https://mintcdn.com/sapient-14d4e375/lnqlKs8p\_hzadcGt/images/favicon.png?fit=max\\u0026auto=format\\u0026n=lnqlKs8p\_hzadcGt\\u0026q=85\\u0026s=358b3be56f25189cd6d5417fd300fefc","background":{"color":{"light":"#FFFFFF","dark":"#0D0C0C"}},"navbar":{"links":\[{"href":"https://lightsage.com","label":"Homepage"}\],"primary":{"type":"button","label":"Dashboard","href":"https://app.lightsage.com"}},"seo":{"indexing":"all"},"fonts":{"family":"Geist"},"integrations":{"ga4":{"measurementId":"G-417S0EJVMS"},"posthog":{"apiKey":"phc\_yrA1TjjqeYfsoJEofjxd0Bw7cTrxs8N402k3NwusII7"}},"contextual":{"options":\["copy","chatgpt","cursor","vscode"\],"display":"header"},"styling":{"latex":false},"navigation":{"pages":\[\]}},"deployment":{"deploymentMetadataValue":{"subdomain":"sapient-14d4e375","hasAudioNarration":false,"actualSubdomain":"sapient-14d4e375","customDomains":\["lightsage.com"\],"gitSource":{"type":"github","owner":"lightsagehq","repo":"docs","deployBranch":"main","isPrivate":true,"contentDirectory":"","mountPath":"","label":""},"chroma":{"liveDeploymentHistoryId":"6ae02c324c09e96410641fcb","collectionId":"82419905-7508-4f98-a30d-d180f437b2ae","schemaVersion":"v3","latestBuiltDeploymentHistoryId":"6ae02c324c09e96410641fcb"},"entitlements":{"AI\_CHAT":{"status":"DISABLED"}},"buildId":"6ae02c324c09e96410641fcb","clientVersion":"0.0.3971","searchFilterCounts":\[{"language":"default","tag":"API Reference","count":74},{"language":"default","tag":"CLI Reference","count":18},{"language":"default","tag":"Evals","count":14},{"language":"default","tag":"Documentation","count":13},{"language":"default","tag":"Prompts","count":13},{"language":"default","tag":"Visibility","count":12},{"language":"default","tag":"MCP Reference","count":10},{"language":"default","tag":"Clients","count":9},{"language":"default","tag":"Agent Preference","count":8},{"language":"default","tag":"Agent Analytics","count":7}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableContextualStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"/docs","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-v7-file-tree","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"pause-universal-search-reconcile","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"per-site-search-delta","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-private-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-nav-generations","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"byok-sep-25","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"vite-origin","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-agent-model-picker","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-updated-9-15","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"isAtlas":false,"googleFontsInlined":true},"customCss":[],"page":{"pagePath":"guides/running-evals","slug":"/guides/running-evals","pageMetadata":{"title":"Running evals","description":"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.","href":"/guides/running-evals"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={code:`code`,p:`p`,pre:`pre`,span:`span`,..._provideComponents(),...props.components},{Card,CardGroup,CodeBlock,CodeGroup,Heading,Note}=_components;if(!Card)_missingMdxReference(`Card`,true);if(!CardGroup)_missingMdxReference(`CardGroup`,true);if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!CodeGroup)_missingMdxReference(`CodeGroup`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`Use `,_jsx(_components.code,{children:`lightsage evals run`}),` to create and start an eval in one command. One\nrequest can test multiple prompts across multiple agents and repeat every pair.`]}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[`The number of attempts is `,_jsx(_components.code,{children:`prompts × agents × runs`}),`. One request can create\nat most 100 attempts.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`set-up-the-cli`,children:`Set up the CLI`}),`\n`,_jsx(_components.p,{children:`Authenticate once, then list the reusable resources available to your\norganization:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`5`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`5`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` auth`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` login`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` agents`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` list`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` repositories`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` list`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` clis`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` list`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` mcp-servers`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` list`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`2`,id:`create-and-run-directly`,children:`Create and run directly`}),`\n`,_jsx(_components.p,{children:`Use the direct form for a one-off eval:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`23`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`23`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --name`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Lightsage interface simulation\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --prompt`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Use the Lightsage CLI to create an eval.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --prompt`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Use the Lightsage MCP server to retrieve an eval.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --judge`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"The requested workflow completes successfully.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --persona`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Platform engineer\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --agent`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` codex:gpt-5-4`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --agent`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` claude-code:claude-opus-4-8`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --cli`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"brew install lightsagehq/tools/lightsage\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --mcp`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` '{`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"name\": \"lightsage\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"type\": \"http\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"url\": \"https://mcp.lightsage.com/mcp\\",\`})}),\`\\n\`,\_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"headers\": {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:' \"X-Lightsage-Api-Key\": \"${LIGHTSAGE_API_KEY}\"'})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }'`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --setup-command`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm install\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --setup-command`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm run db:migrate\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --cleanup-command`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm run db:cleanup\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --env`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` LIGHTSAGE_API_KEY=\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`$LIGHTSAGE_API_KEY`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --runs`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --output-format`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` json`})]}),`\n`]})})}),`\n`,_jsx(_components.p,{children:`This command creates eight attempts: two prompts × two agents × two runs. The\npersona applies to every attempt without adding another fan-out dimension.`}),`\n`,_jsxs(_components.p,{children:[`The response contains the persisted `,_jsx(_components.code,{children:`eval_id`}),` and the parent run `,_jsx(_components.code,{children:`id`}),`. Keep the\nrun ID to monitor the full matrix.`]}),`\n`,_jsx(Heading,{level:`3`,id:`attach-multiple-clis-and-mcp-servers`,children:`Attach multiple CLIs and MCP servers`}),`\n`,_jsxs(_components.p,{children:[`Repeat `,_jsx(_components.code,{children:`--cli`}),` and `,_jsx(_components.code,{children:`--mcp`}),` in request order. Each value can reference a saved\nresource or define one inline:`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`10`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`10`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --prompt`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Exercise every available Lightsage interface.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --judge`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"The workflow completes successfully.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --agent`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` codex:gpt-5-4`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --cli`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 230f0c71-ed91-4a4a-9932-9cef267411bc`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --cli`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm install --global @example/tools\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --mcp`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 550e8400-e29b-41d4-a716-446655440000`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --mcp`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` @lightsage-mcp.json`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --env`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` LIGHTSAGE_API_KEY=\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`$LIGHTSAGE_API_KEY`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --runs`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 3`})]}),`\n`]})})}),`\n`,_jsxs(_components.p,{children:[`For `,_jsx(_components.code,{children:`--cli`}),`, a UUID references a saved CLI. Any other value is an installation\ncommand.`]}),`\n`,_jsxs(_components.p,{children:[`For `,_jsx(_components.code,{children:`--mcp`}),`, pass a saved UUID, one inline JSON object, or `,_jsx(_components.code,{children:`@path/to/file.json`}),`.\nAn MCP file can contain either the canonical object with a `,_jsx(_components.code,{children:`name`}),` field or a\none-server MCP config map:`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`9`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`9`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"lightsage\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"type\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"http\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"url\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"https://mcp.lightsage.com/mcp\\"\`}),\_jsx(\_components.span,{style:{color:\`#1F2328\`,\\"--shiki-dark\\":\`#D4D4D4\`},children:\`,\`})\]}),\`\\n\`,\_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"headers\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"X-Lightsage-Api-Key\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:'\"${LIGHTSAGE_API_KEY}\"'})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),`\n`,_jsxs(_components.p,{children:[`The `,_jsx(_components.code,{children:\"${LIGHTSAGE_API_KEY}\"}),` reference is resolved from repeated `,_jsx(_components.code,{children:`--env KEY=VALUE`}),`\nfields when the sandbox starts. Lightsage does not return or add the resolved\nvalue to public run metadata.`]}),`\n`,_jsx(_components.p,{children:`Saved MCP IDs and inline MCP definitions use the same runtime path after\nresolution. Lightsage translates the portable MCP definition into the native\nconfiguration for Claude Code, Codex, Cursor, GitHub Copilot, Antigravity,\nGemini CLI, OpenClaw, OpenCode, Hermes, Pi, Kilo, and Grok.`}),`\n`,_jsxs(_components.p,{children:[`Use repeatable `,_jsx(_components.code,{children:`--skill owner/repository`}),` values to install public agent skills\nbefore the selected harness starts. Skill installation and both saved and\ninline CLI installation commands run during the same pre-execution setup.`]}),`\n`,_jsx(Heading,{level:`3`,id:`run-setup-and-cleanup-commands`,children:`Run setup and cleanup commands`}),`\n`,_jsxs(_components.p,{children:[`Repeat `,_jsx(_components.code,{children:`--setup-command`}),` to run repository-specific preparation after CLI and\nskill installation but before the agent starts. Setup commands run in order in\nthe prepared repository. A failed setup command stops the attempt before agent\nexecution.`]}),`\n`,_jsxs(_components.p,{children:[`Repeat `,_jsx(_components.code,{children:`--cleanup-command`}),` for best-effort teardown after the agent result is\ncaptured and before the sandbox is deleted. Cleanup runs for successful and\nfailed agent attempts. If one cleanup command fails, Lightsage records the\nfailure and continues with the remaining cleanup commands without replacing\nthe attempt’s primary result.`]}),`\n`,_jsx(Heading,{level:`3`,id:`prompt-an-agent-to-run-the-eval`,children:`Prompt an agent to run the eval`}),`\n`,_jsx(_components.p,{children:`Use this prompt when you delegate the workflow to a coding agent:`}),`\n`,_jsx(CodeGroup,{children:_jsx(CodeBlock,{filename:`Lightsage CLI`,numberOfLines:`8`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`8`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Use the Lightsage CLI to directly create and run a one-off eval with`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:\"`lightsage evals run`.\"})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Create a simple eval about crawling a public webpage about fish. Use`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:\"`codex:gpt-5-4`, include a clear success criterion, and run it once. Return the\"})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`created eval ID and run ID. Monitor the run until it reaches a terminal state,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`then retrieve its analysis.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})})}),`\n`,_jsx(Heading,{level:`2`,id:`create-a-reusable-eval`,children:`Create a reusable eval`}),`\n`,_jsx(_components.p,{children:`Save the simulation matrix when you expect to run it again:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`12`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`12`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` create`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --name`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"SDK installation\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --prompt`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Install the SDK and send a successful request.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --judge`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"The SDK is installed and the request succeeds.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --persona`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Application developer\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --agent`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` codex:gpt-5-4`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --cli`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"brew install lightsagehq/tools/lightsage\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --mcp`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` @lightsage-mcp.json`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --setup-command`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm install\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --cleanup-command`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm run db:cleanup\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --runs`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --tag`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` onboarding`})]}),`\n`]})})}),`\n`,_jsx(_components.p,{children:`Saved evals can contain prompts, judges, agents, persona, CLIs, MCP servers,\nsetup commands, cleanup commands, repeat count, and tags. They do not store\ninline secret values.`}),`\n`,_jsx(_components.p,{children:`Create a saved configuration only when the eval also needs a reusable\nrepository, environment profile, skill set, or legacy resource selection:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`7`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`7`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` configurations`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` create`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --eval-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` eval_123`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --name`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Configured runtime\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --repository-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` repo_123`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --environment-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` env_123`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --skill-ids`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` skill_123`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --skills`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` resend/resend-skills`})]}),`\n`]})})}),`\n`,_jsx(_components.p,{children:`Run the saved defaults:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`4`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`4`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --eval-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` eval_123`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --configuration-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` config_123`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --env`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` LIGHTSAGE_API_KEY=\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`$LIGHTSAGE_API_KEY`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`})]}),`\n`]})})}),`\n`,_jsx(_components.p,{children:`The configuration is optional. You can also override the saved prompts,\npersona, agents, CLIs, MCP servers, setup commands, cleanup commands, or repeat\ncount for one launch:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`7`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`7`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --eval-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` eval_123`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --prompt`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Run this narrower scenario.\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --agent`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` codex:gpt-5-4`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --setup-command`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm install\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --cleanup-command`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"npm run db:cleanup\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --runs`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 5`})]}),`\n`]})})}),`\n`,_jsxs(_components.p,{children:[`In the public API, send an explicit empty `,_jsx(_components.code,{children:`setup_commands`}),`, `,_jsx(_components.code,{children:`cleanup_commands`}),`,\n`,_jsx(_components.code,{children:`clis`}),`, or `,_jsx(_components.code,{children:`mcps`}),` array to disable the saved collection for one launch.`]}),`\n`,_jsx(Heading,{level:`2`,id:`monitor-progress`,children:`Monitor progress`}),`\n`,_jsx(_components.p,{children:`Monitor all active eval runs:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` monitor`})]}),`\n`]})})}),`\n`,_jsx(_components.p,{children:`Monitor one run or retrieve a finite status response:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`2`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` monitor`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --run-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run_123`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` eval-runs`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` retrieve`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --run-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run_123`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --output-format`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` json`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`2`,id:`inspect-analysis`,children:`Inspect analysis`}),`\n`,_jsx(_components.p,{children:`After the run reaches a terminal state, retrieve its persisted judge analysis:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`lightsage`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` evals`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` analysis`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --run-id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run_123`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --output-format`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` json`})]}),`\n`]})})}),`\n`,_jsx(_components.p,{children:`Each attempt includes its prompt, prompt index, agent, and repeat index so you\ncan compare every matrix cell.`}),`\n`,_jsx(Heading,{level:`2`,id:`related-reference`,children:`Related reference`}),`\n`,_jsxs(CardGroup,{cols:2,children:[_jsx(Card,{title:`Evals`,icon:`flask-vial`,href:`/cli-reference/evals`,children:_jsx(_components.p,{children:`Create reusable simulation definitions and configurations.`})}),_jsx(Card,{title:`Eval runs`,icon:`circle-play`,href:`/cli-reference/eval-runs`,children:_jsx(_components.p,{children:`Start runs and monitor their progress.`})}),_jsx(Card,{title:`CLIs`,icon:`terminal`,href:`/cli-reference/clis`,children:_jsx(_components.p,{children:`Find saved CLI IDs that can be attached to an eval.`})}),_jsx(Card,{title:`MCP servers`,icon:`server`,href:`/cli-reference/mcp-servers`,children:_jsx(_components.p,{children:`Find saved MCP server IDs that can be attached to an eval.`})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Running evals","description":"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.","href":"/guides/running-evals"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Set up the CLI","slug":"set-up-the-cli","depth":2,"children":[]},{"title":"Create and run directly","slug":"create-and-run-directly","depth":2,"children":[{"title":"Attach multiple CLIs and MCP servers","slug":"attach-multiple-clis-and-mcp-servers","depth":3,"children":[]},{"title":"Run setup and cleanup commands","slug":"run-setup-and-cleanup-commands","depth":3,"children":[]},{"title":"Prompt an agent to run the eval","slug":"prompt-an-agent-to-run-the-eval","depth":3,"children":[]}]},{"title":"Create a reusable eval","slug":"create-a-reusable-eval","depth":2,"children":[]},{"title":"Monitor progress","slug":"monitor-progress","depth":2,"children":[]},{"title":"Inspect analysis","slug":"inspect-analysis","depth":2,"children":[]},{"title":"Related reference","slug":"related-reference","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"sapient-14d4e375","siteName":"Lightsage","title":"Running evals - Lightsage","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-09-15T23:54:24.816Z","originalFileLocation":"guides/running-evals.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.\">\u003cmeta data-mint-head name=\"application-name\" content=\"Lightsage\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Running evals - Lightsage\">\u003cmeta data-mint-head property=\"og:description\" content=\"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Lightsage\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://lightsage.com/docs/guides/running-evals\\">\\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://sapient-14d4e375.mintlify.app/\_mintlify/api/og/v2?division=Guides\\u0026amp;title=Running+evals\\u0026amp;description=Create%2C+fan+out%2C+monitor%2C+and+inspect+coding-agent+evals+with+the+Lightsage+CLI.\\u0026amp;theme=dd8ce0054346ff07fc865000\\u0026amp;signature=ae36a72fb534259266a6ab619e25f7a1d0319285b12b06855d5dbf4c7506aacb\\">\\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Running evals - Lightsage\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://sapient-14d4e375.mintlify.app/\_mintlify/api/og/v2?division=Guides\\u0026amp;title=Running+evals\\u0026amp;description=Create%2C+fan+out%2C+monitor%2C+and+inspect+coding-agent+evals+with+the+Lightsage+CLI.\\u0026amp;theme=dd8ce0054346ff07fc865000\\u0026amp;signature=ae36a72fb534259266a6ab619e25f7a1d0319285b12b06855d5dbf4c7506aacb\\">\\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/docs/mintlify-assets/_mintlify/favicons/sapient-14d4e375/dqjrUzdCO-t6TId6/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Lightsage\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#0EA5E9\">\u003clink data-mint-head rel=\"canonical\" href=\"https://lightsage.com/docs/guides/running-evals\\">\\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/docs/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/docs/guides/running-evals.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\\",\\"@graph\\":\[{\\"@type\\":\\"Organization\\",\\"@id\\":\\"https://lightsage.com/#organization\\",\\"name\\":\\"Lightsage\\",\\"url\\":\\"https://lightsage.com\\",\\"logo\\":{\\"@type\\":\\"ImageObject\\",\\"url\\":\\"https://mintcdn.com/sapient-14d4e375/0RQiSxJPQv6MyuZt/images/lightsage-logo-with-text-light.svg?fit=max\\u0026auto=format\\u0026n=0RQiSxJPQv6MyuZt\\u0026q=85\\u0026s=843d0563baa54c778891f942711ab938\\"}},{\\"@type\\":\\"WebSite\\",\\"@id\\":\\"https://lightsage.com/docs#website\\",\\"name\\":\\"Lightsage\\",\\"url\\":\\"https://lightsage.com/docs\\",\\"publisher\\":{\\"@id\\":\\"https://lightsage.com/#organization\\"}},{\\"@type\\":\\"WebPage\\",\\"@id\\":\\"https://lightsage.com/docs/guides/running-evals#webpage\\",\\"url\\":\\"https://lightsage.com/docs/guides/running-evals\\",\\"name\\":\\"Running evals\",\"description\":\"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.\",\"dateModified\":\"2026-09-15T23:54:24.816Z\",\"isPartOf\":{\"@id\":\"https://lightsage.com/docs#website\\"},\\"breadcrumb\\":{\\"@id\\":\\"https://lightsage.com/docs/guides/running-evals#breadcrumb\\"}},{\\"@type\\":\\"BreadcrumbList\\",\\"@id\\":\\"https://lightsage.com/docs/guides/running-evals#breadcrumb\\",\\"itemListElement\\":\[{\\"@type\\":\\"ListItem\\",\\"position\\":1,\\"name\\":\\"Guides\\",\\"item\\":\\"https://lightsage.com/docs/workflows/pull-actionables-into-your-agent\\"},{\\"@type\\":\\"ListItem\\",\\"position\\":2,\\"name\\":\\"Running evals\",\"item\":\"https://lightsage.com/docs/guides/running-evals\\"}\]},{\\"@type\\":\[\\"Article\\",\\"TechArticle\\"\],\\"@id\\":\\"https://lightsage.com/docs/guides/running-evals#article\\",\\"headline\\":\\"Running evals\",\"name\":\"Running evals\",\"description\":\"Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.\",\"url\":\"https://lightsage.com/docs/guides/running-evals\\",\\"mainEntityOfPage\\":{\\"@id\\":\\"https://lightsage.com/docs/guides/running-evals#webpage\\"},\\"dateModified\\":\\"2026-09-15T23:54:24.816Z\\",\\"publisher\\":{\\"@id\\":\\"https://lightsage.com/#organization\\"},\\"isPartOf\\":{\\"@id\\":\\"https://lightsage.com/docs#website\\"}}\]}\\u003c/script>\\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\\",\\"@type\\":\\"WebSite\\",\\"name\\":\\"Lightsage\\",\\"creator\\":{\\"@type\\":\\"Organization\\",\\"name\\":\\"Mintlify\\",\\"url\\":\\"https://mintlify.com\\"}}\\u003c/script>","siteName":"Lightsage","nav":{"global":null,"tabs":\[{"tab":"Documentation","pages":\[{"group":"Quickstart","pages":\[{"title":"Quickstart","sidebarTitle":"Quickstart","href":""},{"title":"Migrate from Sapient to Lightsage","sidebarTitle":"Migrate from Sapient","icon":"arrows-rotate","href":"/migration-guide"},{"group":"Onboard your AI agent","root":{"title":"Onboard your AI agent to Lightsage","sidebarTitle":"Onboard your AI agent","href":"/agent-onboarding"},"expanded":true,"pages":[{"title":"Agent Skills","href":"/agent-onboarding/agent-skills"},{"title":"Lightsage MCP","href":"/agent-onboarding/lightsage-mcp"},{"title":"Lightsage CLI","href":"/agent-onboarding/lightsage-cli"},{"title":"Workflows","href":"/agent-onboarding/workflows"},{"title":"Docs for agents","href":"/agent-onboarding/docs-for-agents"}]}]},{"group":"Guides","pages":[{"title":"Pull actionables into your agent","description":"Copy a prompt that has your coding agent pull your Lightsage tasks and opportunities and ship fixes for them.","href":"/workflows/pull-actionables-into-your-agent"},{"title":"Running evals","href":"/guides/running-evals"}]},{"group":"Agent Analytics","pages":[{"title":"Measure agent-attributed growth","description":"Connect visits from recognized AI agent sources to sign-up, demo, checkout, and activation milestones in Agent Analytics.","sidebarTitle":"Attributed growth","href":"/agent-analytics/attributed-growth"}]},{"group":"Integrations","pages":[{"title":"Google Cloud DNS integration","sidebarTitle":"Google Cloud DNS","href":"/agent-analytics/integrations/google-cloud-dns"}]},{"group":"Lightsage Docs Agent","pages":[{"title":"Lightsage Docs Agent","sidebarTitle":"Overview","href":"/lightsage-docs-agent/overview"},{"title":"Run docs evals from GitHub","sidebarTitle":"GitHub PR evals","href":"/lightsage-docs-agent/github-pr-evals"}]}]},{"tab":"CLI Reference","pages":[{"title":"CLI Reference","href":"/cli-reference"}]},{"tab":"MCP Reference","pages":[{"title":"MCP Reference","href":"/mcp-reference"}]},{"tab":"API Reference","pages":[{"title":"API Reference","href":"/api-reference/introduction"}]},{"tab":"Changelog","pages":[{"title":"Changelog","href":"/changelog"}]}]},"title":"Running evals - Lightsage","buildId":"dpl_GvhK4qVXq51DpfYJLCtPpYNN4YHN","deploymentId":"dpl_GvhK4qVXq51DpfYJLCtPpYNN4YHN","dynamics":[],"chunks":[]}