Run docs evals from GitHub - Lightsage

Run docs evals from GitHub

Use GitHub PR evals to test documentation changes against a staging docs URL before merging. Lightsage runs the configured custom evals for the connected docs source and stores the results in your eval history.

Commands

Put the command on the first non-empty line of a pull request comment.

Run evals

1. Confirm the pull request has a staging docs URL The URL can appear in the PR body, a PR comment, or a deployment attached to the pull request.

2. Comment on the pull request
   Add the run command:

   /lightsage run evals
   

3. Wait for Lightsage to reply Lightsage posts a comment when the evals are queued or when it needs more information.

4. Review the eval history In Lightsage, open Custom evals and filter the history source to PR bot.

Include the docs URL in the command comment

If the pull request has more than one preview URL, include the docs URL in the same comment as the command:

/lightsage run evals
Docs preview: https://example-docs-preview.vercel.app

Lightsage uses that URL for the eval run.

How Lightsage finds the docs URL

Lightsage searches the pull request context for a docs URL. It checks:

• The /lightsage run evals comment.
• The pull request body.
• Other pull request comments.
• Deployments attached to the pull request.

If Lightsage cannot find a docs URL, it does not queue evals. It posts a comment explaining that a staging docs URL is required.

Permissions

The comment author must have write, maintain, or admin access to the repository. This prevents drive-by comments from running evals and using credits. The repository also needs the Lightsage GitHub App installed and connected to a Lightsage docs source.

Check status

Use /lightsage status to see the latest Lightsage eval activity for the pull request:

/lightsage status

Lightsage replies with the latest queued or running eval status it can find for that PR.

Troubleshooting

• Lightsage cannot find a docs URL
  Add the staging docs URL to the /lightsage run evals comment, the PR body, or a PR comment. Then run the command again.

• Lightsage says the command is not authorized
  Ask someone with write, maintain, or admin access to run the command, or update the comment author’s repository permissions.

• No evals are configured
  Configure custom evals for the connected docs source in Lightsage, then run the command again.

• The run used the wrong preview URL
  Run the command again and include the intended URL directly in the command comment.