api performance.md
API Performance: Benchmark How Agents Use Your API
Benchmark how AI coding agents discover, call, and execute your API endpoints. See letter grades, per-endpoint pass rates, and actionable fixes across agents.
API Performance measures what happens after an AI coding agent decides to use your API. It runs real agents — Claude Code, Codex, Cursor, and others — through tasks that involve your endpoints, then scores the results across six metrics. The output is a letter grade for your API and a per-endpoint breakdown showing exactly where agents succeed and where they fail.
What API Performance evaluates
Unlike Visibility Analytics, which tracks whether AI recommends you, API Performance measures whether agents can successfully use your API once they've chosen it. This covers the full flow:
- Can the agent find and understand your documentation?
- Can it write correct code to call your endpoints?
- Does that code actually work when executed?
- Can it recover when something goes wrong?
Each evaluation run sends a real coding agent through a defined task and captures the results at every step.
The six metrics
Every API Performance run scores six metrics for each endpoint and agent combination:
| Metric | What it measures | Unit |
|---|---|---|
| Eval Score | Overall composite performance score for the endpoint across all sub-metrics | 0–100 |
| Discovery Score | How reliably the agent locates, reads, and understands your endpoint documentation before attempting to call it | 0–100 |
| Cost per run | Average token cost of the agent completing a task using this endpoint | USD |
| Tool calls | Number of API calls the agent makes to complete the task — fewer is better | Count |
| Errors | Number of errors encountered during the task — includes auth failures, validation errors, and 5xx responses | Count |
| Time to complete | End-to-end time from task start to successful completion | Seconds |
Note
Eval Score and Discovery Score are composite indexes. Eval Score factors in success rate, error count, and recovery capability. Discovery Score measures whether the agent correctly identifies your endpoint, authentication method, and request shape without hallucinating parameters.
Letter grades
Lightsage assigns a letter grade to each endpoint and to your API overall based on the aggregated Eval Score across all runs:
| Grade | Score range | What it means |
|---|---|---|
| A | 90–100 | Agents reliably discover and use this endpoint. Generated code works on the first or second attempt with minimal errors. |
| B | 75–89 | Agents succeed most of the time but encounter occasional discovery or execution issues. Worth improving but not blocking. |
| C | 60–74 | Agents struggle noticeably. Either discovery is inconsistent or error rates are high enough that agents sometimes switch to a competitor. |
| D | Below 60 | Significant usability problems. Agents frequently fail or abandon this endpoint. High likelihood of generating frustrated developers at scale. |
Leaderboard benchmarks
Lightsage's public Devtool Arena provides reference points for how your scores compare to other APIs in your category. Sample data from the Payments category:
| API | Overall score | Grade |
|---|---|---|
| Stripe | 82 | B |
| PayPal | 81 | B |
| Razorpay | 69 | C |
| Paddle | 63 | C |
| Chargebee | 65 | C |
These scores reflect real evaluation runs across multiple coding agents on the Devtool Arena leaderboard. A score of 82 (Stripe's position) means agents succeed on most tasks but still encounter some friction — there's measurable room for improvement even for a top performer in the category.
Configuring evaluations
1. Add your endpoints
Go to API Performance in your dashboard and select Add Endpoints. Paste your endpoint URLs or import from an OpenAPI spec. Add the authentication method and any required headers.
2. Choose coding agents
Select which agents to run evaluations against. Each agent costs 5 credits per run. Start with Claude Code and Codex — they're the most widely used and provide the clearest signal. Add more agents as your credit budget allows.
3. Set evaluation frequency
Choose daily or weekly evaluation runs. Daily is recommended when you're actively improving documentation or error messages. Weekly works well for stable APIs where you want trend data without high credit consumption.
4. Review baseline results
After the first run, review your scores in the endpoint breakdown. The baseline tells you where to focus — start with D- and C-grade endpoints, since improvements there have the highest impact on overall grade.
Endpoint-level breakdown
The endpoint breakdown shows pass rates for each of your endpoints across all selected coding agents. Example from the Stripe Payments API evaluation:
| Endpoint | Pass rate | Grade |
|---|---|---|
POST /v1/payment_intents |
94% | A |
POST /v1/charges |
88% | A |
GET /v1/customers/{id} |
81% | B |
POST /v1/refunds |
72% | C |
POST /v1/disputes/{id}/close |
63% | C |
Pass rate is the percentage of agent runs where the task completed successfully — meaning the agent called the endpoint, received a valid response, and the overall task finished without switching to a competitor.
Endpoints with low pass rates are your highest-priority fixes. A pass rate below 70% means a meaningful percentage of developers who ask a coding agent to use that endpoint will hit a dead end.
Coding agent leaderboard
The agent leaderboard shows which coding agents work best with your API — useful for understanding where to focus optimization effort and for communicating compatibility to your users.
Sample leaderboard data across five endpoint categories:
| Agent | Average | Category 1 | Category 2 | Category 3 | Category 4 | Category 5 |
|---|---|---|---|---|---|---|
| Claude Code | 94% | 97% | 96% | 92% | 90% | 95% |
| Codex | 87% | 91% | 89% | 85% | 81% | 88% |
| Cursor | 76% | 82% | 78% | 72% | 68% | 80% |
| GitHub Copilot | 71% | 78% | 74% | 70% | 61% | 72% |
| Hermes | 63% | 71% | 65% | 58% | 56% | 63% |
| OpenCode | 58% | 66% | 60% | 54% | 51% | 59% |
A large gap between Claude Code and other agents usually indicates that your documentation is written in a style that Claude's model understands well, but that other agent architectures find ambiguous. Consistent response shapes and descriptive errors narrow those gaps.
The Agent Journey flow
Lightsage records each agent's navigation path as it works through a task. The Agent Journey flow visualizes how agents move from your documentation to your API endpoints.
Each successful path shows:
- Steps taken — how many pages or tool calls the agent made before reaching a successful API call
- Starting point — where the agent began (usually a docs page or search result)
- Entry point — which documentation page the agent read before calling the endpoint
Paths with many steps signal that agents are struggling to find the right endpoint or understand the request format. If an agent consistently takes 6+ steps to call a simple endpoint, your documentation structure or endpoint naming is causing unnecessary navigation.
Interpreting low scores and what to fix
Low Discovery Score
The agent can't reliably find or understand your endpoint. Common causes:
- AI crawlers are blocked in
robots.txt— ClaudeBot or GPTBot can't read your docs - Documentation is client-side rendered JavaScript and doesn't work without a browser
- Endpoint names are ambiguous — agents can't determine which endpoint handles a given task
- No
llms.txtfile — agents have no machine-readable summary of your API's capabilities
Fix: Check your robots.txt, verify server-side rendering on your docs, and add an llms.txt file at your domain root.
High error count
Agents frequently hit errors when calling your endpoint. Common causes:
- Authentication setup is unclear — agents guess at header names or token formats
- Error messages are generic (
Error 500,Bad Request) with no guidance on how to fix them - Required parameters are poorly documented — agents omit them or use wrong types
- Breaking API changes aren't reflected in current documentation
Fix: Add descriptive error messages that explain what went wrong and what to change. Audit your most-called endpoints for parameter documentation completeness.
High tool call count
Agents are making many API calls to complete a single task — a sign they're exploring, recovering from failures, or confused about the right approach.
Fix: Add a clear quick-start example for the endpoint that shows the exact code an agent should generate. Agents with a concrete example to pattern-match against make fewer exploratory calls.
Low completion rate / agents switching
Agents start with your API but switch to a competitor before finishing the task. This is the most damaging failure mode — it generates a negative experience at scale.
Fix: Review the Agent Journey flow for abandoned runs. Identify the last endpoint called before the agent switched — that's your highest-priority fix. Usually this is an unclear error message or a missing response field.
Warning
High visibility + low API Performance is worse than low visibility. If coding agents recommend your API frequently but generated code fails, you're generating frustrated developers at scale. Fix API Performance issues before investing heavily in visibility improvements.
Credit costs for API Performance
API Performance evaluations use coding agents, which cost 5 credits per run. A single endpoint evaluated against one agent once costs 5 credits. Evaluate 5 endpoints against 3 agents on a weekly schedule and you'll use approximately 300 credits per month.
Use the credit estimator on the pricing page to plan your evaluation budget before enabling multiple agents or daily runs.