## Run docs evals from GitHub

Use GitHub PR evals to test documentation changes against a staging docs URL before merging. Lightsage runs the configured custom evals for the connected docs source and stores the results in your eval history.

## Commands

| Command                    | What it does                                         |
|---------------------------|-----------------------------------------------------|
| `/lightsage run evals`    | Queues docs evals for the pull request.             |
| `/lightsage status`        | Shows the latest Lightsage eval status for the pull request. |
| `/lightsage`               | Shows available Lightsage commands.                  |
| `/lightsage help`         | Shows available Lightsage commands.                  |

Put the command on the first non-empty line of a pull request comment.

## Run evals

1. Confirm the pull request has a staging docs URL 
   The URL can appear in the PR body, a PR comment, or a deployment attached to the pull request.

2. Comment on the pull request  
   Add the run command:
   ```
   /lightsage run evals
   ```

3. Wait for Lightsage to reply
   Lightsage posts a comment when the evals are queued or when it needs more information.

4. Review the eval history
   In Lightsage, open **Custom evals** and filter the history source to **PR bot**.

## Include the docs URL in the command comment

If the pull request has more than one preview URL, include the docs URL in the same comment as the command:
``` 
/lightsage run evals
Docs preview: https://example-docs-preview.vercel.app
```
Lightsage uses that URL for the eval run.

## How Lightsage finds the docs URL

Lightsage searches the pull request context for a docs URL. It checks:
- The `/lightsage run evals` comment.
- The pull request body.
- Other pull request comments.
- Deployments attached to the pull request.

If Lightsage cannot find a docs URL, it does not queue evals. It posts a comment explaining that a staging docs URL is required.

## Permissions

The comment author must have write, maintain, or admin access to the repository. This prevents drive-by comments from running evals and using credits. The repository also needs the Lightsage GitHub App installed and connected to a Lightsage docs source.

## Check status

Use `/lightsage status` to see the latest Lightsage eval activity for the pull request:
``` 
/lightsage status
```
Lightsage replies with the latest queued or running eval status it can find for that PR.

## Troubleshooting

- **Lightsage cannot find a docs URL**  
  Add the staging docs URL to the `/lightsage run evals` comment, the PR body, or a PR comment. Then run the command again.

- **Lightsage says the command is not authorized**  
  Ask someone with write, maintain, or admin access to run the command, or update the comment author’s repository permissions.

- **No evals are configured**  
  Configure custom evals for the connected docs source in Lightsage, then run the command again.

- **The run used the wrong preview URL**  
  Run the command again and include the intended URL directly in the command comment.