Submit your API to Devtool Arena for evaluation

Submitting your API to Devtool Arena means Lightsage evaluates your tool using real, live AI coding agent runs — not synthetic benchmarks. Claude Code receives a production API key, a real developer task specific to your category, and a fresh environment. Lightsage records what happens: whether the agent succeeded, how many tool calls it made, what errors it hit, and how long it took. The results appear on the public leaderboard so you and your team can track your ranking over time.

What you need to submit

To request an evaluation, go to lightsage.com/leaderboard/add and provide:

• Your documentation URL — the primary URL where your API reference and quick-start guide live, or
• Your GitHub repo URL — if your API is primarily documented there
• Your email address — so Lightsage can reach you with results and questions

That is the complete set of required inputs. You do not need to provide an API key at submission time — Lightsage will locate or request one as part of the evaluation setup.

The submission and evaluation process

1. Submit your documentation URL or GitHub repo
Fill in the form at lightsage.com/leaderboard/add with your docs URL (or GitHub URL) and your work email. You will receive a confirmation once your submission is received.

2. Lightsage obtains credentials
Lightsage signs up for your product and obtains a working API key through your standard developer onboarding flow — the same path any developer would take. If your API requires a paid plan, company verification, or a credit card to access, your evaluation may be queued as Skipped until credentials can be obtained. Reach out if you can provide a test key directly.

3. Live evaluation runs
Claude Code receives your API key, your documentation URL, and a realistic developer task. It runs the task end-to-end using live API calls against your production (or sandbox) environment. Lightsage records Eval Score, Discovery Score, tool calls, errors, cost, and time. Multiple runs are averaged for a stable result.

4. Results published and report shared
Your tool appears on the leaderboard. Lightsage notifies you by email with a link to your full eval report, which includes the task prompt used, the agent's execution trace, specific errors encountered, and your scores across all metrics.

5. Claim your company report
Log in at lightsage.com to claim ownership of your company's report. Once claimed, you can view detailed session logs, track score changes over time, and share your eval report with your team.

What the evaluation covers

Each evaluation run uses a real, realistic developer task drawn from your API's primary use case. For example:

• A payment API might be asked to create a customer and charge them
• A voice TTS API might be asked to synthesize speech from a text prompt and save the output
• A vector database might be asked to create a collection, upsert embeddings, and run a similarity query

The agent uses your public documentation to figure out how to complete the task — it does not receive any special guidance beyond what any developer would find on your site. Live API calls are made against your actual API endpoints.

Why some evaluations are skipped

If Lightsage cannot obtain a working API key through your standard developer onboarding, your tool is listed as Skipped on the leaderboard with a reason noted. Common causes:

• API key requires a paid plan — free tier credentials are not issued
• Signup requires a credit card — even for free tiers
• Access requires company or enterprise verification — no self-serve developer access
• Open-source / self-hosted — no hosted API available to test against
• MCP server requires browser-based OAuth — cannot be automated

If your tool was skipped and you want to be evaluated, email the leaderboard team or resubmit with a note explaining how to access credentials. Providing a time-limited test key directly is the fastest path to getting evaluated.

How to maximize your score before submitting

The following improvements directly affect your Eval Score and Discovery Score. Make these changes before you submit for the best first result.

• Allow AI crawlers in robots.txt
  Many APIs block AI crawlers by default, which prevents agents from reading your documentation at evaluation time. Check your robots.txt and ensure you are not blocking common agent user-agents. If you use Cloudflare or another WAF, verify that AI crawler traffic is not being rate-limited or blocked.

• Add an llms.txt file
  Publish an llms.txt file at the root of your documentation domain (e.g., https://docs.yourtool.com/llms.txt). The file should include a one-sentence description of what your API does, a "when to use" section, and a "when NOT to use" section. The "when not to use" section is especially effective — it helps agents make accurate recommendations and reduces over-recommendation in contexts where your API is a poor fit. Include your quick-start endpoint and authentication format.

• Ensure your quick start works end-to-end
  Walk through your own quick-start guide from a fresh environment with no prior knowledge of your API. If any step requires something not documented — a dashboard action, an environment variable not mentioned, a prerequisite library — an agent will get stuck at that step. Fix any gaps so the quick start works with only the information on your docs site.

• Use descriptive, specific error messages
  Error messages are one of the strongest signals of API quality for agents. When the agent hits an error, a specific message dramatically increases recovery rate. Compare:

  • "Invalid API key format. Expected: sk_live_xxx or sk_test_xxx" — 89% agent recovery rate
  • "Authentication failed. Check your API key." — 67% recovery rate
  • "Error 401" — 34% recovery rate

Update your API to return the most specific error context possible, especially for authentication failures, missing parameters, and invalid input shapes.

• Publish TypeScript types or an OpenAPI spec
  Agents generate substantially better code when they can see the shape of your responses. Publish TypeScript type definitions, JSON Schema, or an OpenAPI/Swagger spec for your API. Consistent, typed response envelopes (rather than varying response shapes) increase tool call success rates by approximately 23%.

• Submit to Context7
  Context7 is a documentation index that coding agents query at runtime. Submitting your documentation to Context7 boosts your Discovery Score by giving agents a structured, up-to-date reference they can pull during a session.

• Consider shipping an MCP server
  Tools with working MCP servers consistently dominate their categories on the MCP leaderboard. If your API has a natural fit for MCP — most do — a well-built MCP server gives agents a direct, structured interface rather than requiring them to interpret raw HTTP docs. Most existing MCP servers are minimally functional; a high-quality one creates a clear category advantage.

After your report is published

Once your eval report is live, you can:

• View your full report — the task prompt, agent execution trace, errors, and per-metric scores
• Share your report — use the report URL to share results with your team or publish your ranking publicly
• Track changes over time — the leaderboard updates multiple times per week, so improvements you ship will appear in subsequent evaluation runs
• Compare against competitors — filter the leaderboard by your category to see how you rank relative to similar tools

How scoring works
Understand exactly how Eval Score and Discovery Score are calculated and what moves each metric.

Leaderboard overview
See how to read the leaderboard table and what each column means.