github pr evals.md

Run docs evals from GitHub

Use Lightsage slash commands on GitHub pull requests to run docs evals against staging documentation.

Use GitHub PR evals to test documentation changes against a staging docs URL before merging. Lightsage runs the configured custom evals for the connected docs source and stores the results in your eval history.

Commands

Command	What it does
`/lightsage run evals`	Queues docs evals for the pull request.
`/lightsage status`	Shows the latest Lightsage eval status for the pull request.
`/lightsage`	Shows available Lightsage commands.
`/lightsage help`	Shows available Lightsage commands.

Put the command on the first non-empty line of a pull request comment.

Run evals

1. Confirm the pull request has a staging docs URL

The URL can appear in the PR body, a PR comment, or a deployment attached to the pull request.

2. Comment on the pull request

Add the run command:

/lightsage run evals

3. Wait for Lightsage to reply

Lightsage posts a comment when the evals are queued or when it needs more information.

4. Review the eval history

In Lightsage, open Custom evals and filter the history source to PR bot.

Include the docs URL in the command comment

If the pull request has more than one preview URL, include the docs URL in the same comment as the command:

/lightsage run evals

Docs preview: https://example-docs-preview.vercel.app

Lightsage uses that URL for the eval run.

How Lightsage finds the docs URL

Lightsage searches the pull request context for a docs URL. It checks:

The /lightsage run evals comment.
The pull request body.
Other pull request comments.
Deployments attached to the pull request.

If Lightsage cannot find a docs URL, it does not queue evals. It posts a comment explaining that a staging docs URL is required.

Permissions

The comment author must have write, maintain, or admin access to the repository. This prevents drive-by comments from running evals and using credits.

The repository also needs the Lightsage GitHub App installed and connected to a Lightsage docs source.

Check status

Use /lightsage status to see the latest Lightsage eval activity for the pull request:

/lightsage status

Lightsage replies with the latest queued or running eval status it can find for that PR.

Troubleshooting

Lightsage cannot find a docs URL
Add the staging docs URL to the /lightsage run evals comment, the PR body, or a PR comment. Then run the command again.

Lightsage says the command is not authorized
Ask someone with write, maintain, or admin access to run the command, or update the comment author's repository permissions.

No evals are configured
Configure custom evals for the connected docs source in Lightsage, then run the command again.

The run used the wrong preview URL
Run the command again and include the intended URL directly in the command comment.