200 real questions from a public help center and an open-source codebase, scored head-to-head against Notion, Atlassian Rovo, Claude Code, and Codex by three frontier LLM judges.
Share of decisive judge verdicts won by Falconer, both tests combined · 200 questions · 4 other AI tools on the market · 3 LLM judges
We judged performance against 200 real questions from a public support corpus and the codebase of a popular open-source project. Falconer wins head-to-head against all leading tools.
Can a system answer real customer support questions using only a help center? We pointed each tool at the same 6,221 Wix help articles, then scored it on 100 real customer questions from the public WixQA set.
Can a system answer real engineering questions from live source code? We indexed the apache/spark repository and its markdown docs, then scored each tool on 100 Spark questions from Stack Overflow.
Note: Falconer, Claude Code, and Codex ran at the same thinking-effort level on each test: medium effort on the docs test, high effort on the code test. Notion and Atlassian Rovo do not expose a comparable effort setting.
Falconer wins the majority of decisive verdicts in every matchup.
Win share of decisive judge verdicts (6 per question), scored by the weighted-sum rule. Ties excluded; the full record is shown beside each matchup.
Matchups were scored holistically, by weighted formula, and by strict Pareto rule.
We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.
| Scoring rule | How it decides | Definition of tie | vsNotion | vsAtlassian Rovo | vsClaude Code | vsCodex |
|---|---|---|---|---|---|---|
| Weighted-sumUsed above | Blend the four axes (0.35 faithfulness + 0.35 helpfulness + 0.20 completeness + 0.10 relevance); a side wins if it leads by more than 0.25 on the 0 to 10 scale. | The two scores land within 0.25 of each other. | 70.5% | 88.4% | 52.6% | 62.8% |
| Holistic | Trust the judge's own overall pick of the better answer. | The judge called it a tie. | 70.5% | 87.8% | 55% | 60.2% |
| Pareto | A side wins only if it scores ≥ on all four axes and > on at least one. The strictest rule. | Equal on every quality metric, or mixed (each side better somewhere). | 74% | 94.6% | 51.4% | 52.5% |
Falconer delivers the fastest full answers: 18.5s median, well ahead of every rival.
Time to full answer and to first token, across the full distribution. Lower and tighter is better.
Notion answered 94 of 100 questions; all other systems answered every question.
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 19.6s | 15.3s | 18.5s | 23.3s | 30.5s |
| Notion | 27.5s | 22.3s | 27.1s | 31.2s | 35.8s |
| Atlassian Rovo | 31.2s | 26.5s | 30.4s | 34.6s | 37s |
| Claude Code | 29.4s | 21.1s | 27.3s | 36.2s | 41.9s |
| Codex | 27.3s | 21.1s | 25.9s | 31s | 41.3s |
Notion answered 94 of 100 questions; all other systems answered every question.
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 3.3s | 2.5s | 2.9s | 3.8s | 4.5s |
| Notion | 19.5s | 14s | 18.3s | 23.3s | 30s |
| Atlassian Rovo | 18.4s | 13.8s | 18.9s | 21.4s | 23.7s |
| Claude Code | 2.2s | 1.7s | 2s | 2.3s | 2.9s |
| Codex | 27s | 19.9s | 25.7s | 30.8s | 41.1s |
Top performers show shorter answer length.
The judges never reward length; it's shown here only as context.
Notion answered 94 of 100 questions; all other systems answered every question.
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 1,768 | 1,242 | 1,741 | 2,138 | 2,705 |
| Notion | 2,419 | 1,831 | 2,367 | 2,832 | 3,944 |
| Atlassian Rovo | 5,730 | 5,357 | 6,032 | 6,829 | 7,529 |
| Claude Code | 1,875 | 1,368 | 1,838 | 2,409 | 2,884 |
| Codex | 1,304 | 918 | 1,244 | 1,592 | 2,091 |
Falconer wins the majority of decisive verdicts in every matchup.
Win share of decisive judge verdicts (6 per question), scored by the weighted-sum rule. Ties excluded; the full record is shown beside each matchup.
Matchups were scored holistically, by weighted formula, and by strict Pareto rule.
We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.
| Scoring rule | How it decides | Definition of tie | vsNotion | vsAtlassian Rovo | vsClaude Code | vsCodex |
|---|---|---|---|---|---|---|
| Weighted-sumUsed above | Blend the four axes (0.35 faithfulness + 0.35 helpfulness + 0.20 completeness + 0.10 relevance); a side wins if it leads by more than 0.25 on the 0 to 10 scale. | The two scores land within 0.25 of each other. | 57.7% | 97.1% | 56.1% | 74.2% |
| Holistic | Trust the judge's own overall pick of the better answer. | The judge called it a tie. | 58.8% | 96.2% | 55% | 74.2% |
| Pareto | A side wins only if it scores ≥ on all four axes and > on at least one. The strictest rule. | Equal on every quality metric, or mixed (each side better somewhere). | 61.1% | 97.7% | 55.7% | 75.7% |
On the code test, speed is a near tie: every tool but Codex (72s) returns a full answer in 39 to 45s median.
Time to full answer and to first token, across the full distribution. Lower and tighter is better.
Atlassian Rovo answered 98 of 100 questions; all other systems answered every question.
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 51.3s | 31.9s | 45s | 64.7s | 85.4s |
| Notion | 45.4s | 28.6s | 40s | 56.2s | 69.8s |
| Atlassian Rovo | 49.3s | 33.2s | 42.6s | 58.3s | 78.3s |
| Claude Code | 45.4s | 25.3s | 39.1s | 55.9s | 86.1s |
| Codex | 78.6s | 58.7s | 72s | 99.8s | 120.8s |
Atlassian Rovo answered 98 of 100 questions; all other systems answered every question.
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 19.1s | 5.5s | 15.5s | 26s | 42.1s |
| Notion | 27.4s | 12.4s | 24.4s | 36.9s | 50.9s |
| Atlassian Rovo | 13.7s | 11.8s | 12.7s | 15.1s | 17.9s |
| Claude Code | 2.1s | 1.6s | 1.9s | 2.2s | 2.8s |
| Codex | 55.1s | 10.2s | 55.7s | 98.5s | 120.7s |
Top performers show shorter answer length.
The judges never reward length; it's shown here only as context.
Atlassian Rovo answered 98 of 100 questions; all other systems answered every question.
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 3,295 | 2,592 | 3,205 | 4,031 | 4,764 |
| Notion | 3,846 | 2,942 | 3,655 | 4,734 | 5,542 |
| Atlassian Rovo | 586 | 136 | 228 | 452 | 1,676 |
| Claude Code | 3,067 | 2,451 | 3,036 | 3,733 | 4,411 |
| Codex | 1,870 | 1,398 | 1,802 | 2,283 | 2,895 |
The full dataset is public, including every question, reference answer, assistant response, and judge verdict: FalconerAI/falconer-benchmarks.
Every head-to-head is scored by Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro, in both A/B and B/A orderings to neutralize position bias, for six verdicts per question.
The headline winner is derived from four metric scores with a fixed weighting. Anyone with the per-metric scores gets the same answer, and the lead holds under two other rules too.
Weighted-sum scores within 0.25 of each other count as ties and are excluded from head-to-head win rates.
Every answer is judged head-to-head against Falconer, with the human-written gold answer (the WixQA reference, or the accepted Stack Overflow answer) as the reference both sides are measured against, rather than grading either answer in isolation.
Every system runs sealed to its corpus. Web tools are disabled and verified at runtime, so results reflect retrieval and grounding.
Both question sets are public (the WixQA support corpus and the apache/spark codebase with its Stack Overflow Q&A), at 100 questions each, so anyone can rerun the same benchmark against their own system.
Every answer scored 0 to 10 against the human reference.
Every claim is correct and supported by the source, with no fabricated steps, wrong API names, invented config keys, or made-up numbers, even when the rest of the answer reads well.
Is every claim true to the source?
Concrete and actionable: real UI paths, working code, specific steps and version numbers. Vague guidance and "check the docs" deflections are penalized.
Can the reader act on it directly?
Covers every critical step or fact needed to actually resolve the question. Tangential background does not raise this score.
Did it cover everything needed to finish the job?
The answer is relevant. Sometimes an answer can be factually true, but irrelevant to the problem in question. Tangents, unrequested background, and citation dumps are penalized. Length itself is not.
How much is on-target signal vs. noise?
score = 0.35·faithfulness + 0.35·helpfulness + 0.20·completeness + 0.10·relevanceThe questions come from public, third-party sources: real help-center Q&A from WixQA and Apache Spark questions from Stack Overflow. For Spark, we selected questions by user votes and by whether answering them well requires reading the repository. We published every question, the reference answer, and each assistant’s full response.