# Falconer benchmarks: head-to-head vs Notion, Atlassian Rovo, Claude Code, and Codex

> Falconer outscores leading AI tools on real support and engineering questions. We judged performance against 200 real questions from a public support corpus and the codebase of a popular open-source project. Falconer wins head-to-head against all leading tools.

- Questions: 200 (two public datasets, 100 each)
- Competitors: Notion, Atlassian Rovo, Claude Code, and Codex
- LLM judges: Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro
- Public dataset (every question, reference answer, assistant response, and judge verdict): https://github.com/FalconerAI/falconer-benchmarks

## Overall win rate, both tests combined

Share of decisive judge verdicts won by Falconer.

| Matchup | Falconer win rate | Wins | Losses |
| --- | --- | --- | --- |
| Falconer vs Notion | 64% | 560 | 311 |
| Falconer vs Atlassian Rovo | 93% | 1064 | 83 |
| Falconer vs Claude Code | 54% | 428 | 360 |
| Falconer vs Codex | 68% | 654 | 304 |

## Benchmarks for docs and code

We judged performance against 200 real questions from a public support corpus and the codebase of a popular open-source project. Falconer wins head-to-head against all leading tools.

## Benchmark results

### Wix help center: documentation test

Can a system answer real customer support questions using only a help center? We pointed each tool at the same 6,221 Wix help articles, then scored it on 100 real customer questions from the public WixQA set.

- Persona: A Wix customer asking a real support question (billing, domains, site setup) and needs a solution.
- Corpus: 6,221 articles · 100 questions
- Source: WixQA on Hugging Face (https://huggingface.co/datasets/Wix/WixQA)
- Effort level: medium effort
- Models tested: Falconer (Claude Opus 4.8 · medium effort); Notion (Claude Opus 4.8 · effort undisclosed); Atlassian Rovo (model undisclosed · Think deeper); Claude Code (Claude Opus 4.8 · medium effort); Codex (GPT-5.5 · medium effort)

**On real support questions, Falconer wins against all four tools.**

#### Head-to-head win rate

Falconer wins the majority of decisive verdicts in every matchup.

| Matchup | Falconer win rate | Record (W-L-T) |
| --- | --- | --- |
| Falconer vs Notion | 70.5% | 316-132-116 |
| Falconer vs Atlassian Rovo | 88.4% | 503-66-31 |
| Falconer vs Claude Code | 52.6% | 213-192-195 |
| Falconer vs Codex | 62.8% | 314-186-100 |

#### Three scoring methods

Matchups were scored holistically, by weighted formula, and by strict Pareto rule.

We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.

| Scoring rule | vs Notion | vs Atlassian Rovo | vs Claude Code | vs Codex |
| --- | --- | --- | --- | --- |
| Weighted-sum (headline) | 70.5% | 88.4% | 52.6% | 62.8% |
| Holistic | 70.5% | 87.8% | 55% | 60.2% |
| Pareto | 74% | 94.6% | 51.4% | 52.5% |

#### Speed and length

Falconer delivers the fastest full answers: 18.5s median, well ahead of every rival. Top performers show shorter answer length.

| System | Full answer (median) | First token (median) | Length (median chars) | Coverage |
| --- | --- | --- | --- | --- |
| Falconer | 18.5s | 2.9s | 1,741 | 100/100 |
| Notion | 27.1s | 18.3s | 2,367 | 94/100 |
| Atlassian Rovo | 30.4s | 18.9s | 6,032 | 100/100 |
| Claude Code | 27.3s | 2s | 1,838 | 100/100 |
| Codex | 25.9s | 25.7s | 1,244 | 100/100 |

### Apache Spark: code test

Can a system answer real engineering questions from live source code? We indexed the apache/spark repository and its markdown docs, then scored each tool on 100 Spark questions from Stack Overflow.

- Persona: A data engineer who wants working code, an accurate config key, or a root cause.
- Corpus: apache/spark code package with markdown files · 100 questions
- Source: apache/spark on GitHub (https://github.com/apache/spark)
- Effort level: high effort
- Models tested: Falconer (Claude Opus 4.8 · high effort); Notion (Claude Opus 4.8 · effort undisclosed); Atlassian Rovo (model undisclosed · Think deeper); Claude Code (Claude Opus 4.8 · high effort); Codex (GPT-5.5 · high effort)

**On real engineering questions, Falconer wins against all four tools.**

#### Head-to-head win rate

Falconer wins the majority of decisive verdicts in every matchup.

| Matchup | Falconer win rate | Record (W-L-T) |
| --- | --- | --- |
| Falconer vs Notion | 57.7% | 244-179-177 |
| Falconer vs Atlassian Rovo | 97.1% | 561-17-10 |
| Falconer vs Claude Code | 56.1% | 215-168-217 |
| Falconer vs Codex | 74.2% | 340-118-141 |

#### Three scoring methods

Matchups were scored holistically, by weighted formula, and by strict Pareto rule.

We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.

| Scoring rule | vs Notion | vs Atlassian Rovo | vs Claude Code | vs Codex |
| --- | --- | --- | --- | --- |
| Weighted-sum (headline) | 57.7% | 97.1% | 56.1% | 74.2% |
| Holistic | 58.8% | 96.2% | 55% | 74.2% |
| Pareto | 61.1% | 97.7% | 55.7% | 75.7% |

#### Speed and length

On the code test, speed is a near tie: every tool but Codex (72s) returns a full answer in 39 to 45s median. Top performers show shorter answer length.

| System | Full answer (median) | First token (median) | Length (median chars) | Coverage |
| --- | --- | --- | --- | --- |
| Falconer | 45s | 15.5s | 3,205 | 100/100 |
| Notion | 40s | 24.4s | 3,655 | 100/100 |
| Atlassian Rovo | 42.6s | 12.7s | 228 | 98/100 |
| Claude Code | 39.1s | 1.9s | 3,036 | 100/100 |
| Codex | 72s | 55.7s | 1,802 | 100/100 |

## How the judges decided

- **Three frontier judges:** Every head-to-head is scored by Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro, in both A/B and B/A orderings to neutralize position bias, for six verdicts per question.
- **A reproducible formula:** The headline winner is derived from four metric scores with a fixed weighting. Anyone with the per-metric scores gets the same answer, and the lead holds under two other rules too.
- **What counts as a tie:** Weighted-sum scores within 0.25 of each other count as ties and are excluded from head-to-head win rates.
- **Pairwise, against a human reference:** Every answer is judged head-to-head against Falconer, with the human-written gold answer (the WixQA reference, or the accepted Stack Overflow answer) as the reference both sides are measured against, rather than grading either answer in isolation.
- **No web access for any agent:** Every system runs sealed to its corpus. Web tools are disabled and verified at runtime, so results reflect retrieval and grounding.
- **Public, reproducible corpora:** Both question sets are public (the WixQA support corpus and the apache/spark codebase with its Stack Overflow Q&A), at 100 questions each, so anyone can rerun the same benchmark against their own system.

## How we measure quality

Every answer scored 0 to 10 against the human reference.

- **Faithfulness (35%, correctness):** Every claim is correct and supported by the source, with no fabricated steps, wrong API names, invented config keys, or made-up numbers, even when the rest of the answer reads well. Signal: Is every claim true to the source?
- **Helpfulness (35%, actionability):** Concrete and actionable: real UI paths, working code, specific steps and version numbers. Vague guidance and "check the docs" deflections are penalized. Signal: Can the reader act on it directly?
- **Completeness (20%, recall):** Covers every critical step or fact needed to actually resolve the question. Tangential background does not raise this score. Signal: Did it cover everything needed to finish the job?
- **Relevance (10%, precision):** The answer is relevant. Sometimes an answer can be factually true, but irrelevant to the problem in question. Tangents, unrequested background, and citation dumps are penalized. Length itself is not. Signal: How much is on-target signal vs. noise?

Weighting formula: `score = 0.35·faithfulness + 0.35·helpfulness + 0.20·completeness + 0.10·relevance`

- Correct and actionable matter most, so faithfulness and helpfulness get 35% each. A fluent answer with the wrong API name is worthless.
- Completeness (20%) catches answers that skip a critical step.
- Relevance (10%) penalizes off-topic padding.
- Completeness and relevance deliberately pull against each other: adding content lifts completeness but costs relevance, so neither rewards bloat.

## Real questions used

The questions come from public, third-party sources: real help-center Q&A from WixQA and Apache Spark questions from Stack Overflow. For Spark, we selected questions by user votes and by whether answering them well requires reading the repository. We published every question, the reference answer, and each assistant’s full response.

## Caveats

- The displayed records use the headline weighted-sum rule: a verdict is a tie when the two weighted scores land within 0.25 of each other on the 0 to 10 scale, too close for the judge to separate (the scoring table shows how each of the three rules defines a tie). Head-to-head % counts wins among the decisive (non-tie) verdicts, the standard way win rates are reported.
- Each question set is 100 items drawn from a public benchmark; larger samples tighten the confidence intervals.
- Coverage varies slightly. Notion answered 94 of 100 support questions; on the other 6 it returned an interactive clarification form instead of an answer, so those are excluded. Rovo answered 98 of 100 engineering questions. Win/loss/tie counts are tallied over the questions each system actually answered (6 verdicts per answered question), so they total below 600 for those systems. One Codex verdict on the engineering test was an unparseable judge response and is also excluded, so that record totals 599.
- On the engineering test, Atlassian Rovo is a structural baseline rather than a like-for-like rival: the apache/spark repository was hosted in Bitbucket, which Rovo barely reads, so its short answers reflect missing source access rather than answer quality. Read that matchup as retrieval-with-code vs retrieval-without-code; Notion, Claude Code, and Codex are the code-capable cohort.
- Per-metric averages and head-to-head wins can diverge: a system can post a higher average on one metric yet win fewer questions, because each question’s winner is decided from all four metrics together.
- Ties (weighted-sum scores within 0.25 of each other) are excluded from head-to-head %. Tie rates ranged from ~2% (Rovo) to ~36% (Claude Code); closer matchups naturally produce more ties.
- The Stack Overflow reference is one valid solution, not the only correct answer; judges credit alternate-but-correct approaches.
- Falconer, Claude Code, and Codex run named models at the same thinking-effort level on each test: medium effort on the support test, high effort on the engineering test. Notion exposes its model (Opus 4.8) but not its effort level; Rovo ran in its "Think deeper" mode on an undisclosed model.
- Results are a point-in-time snapshot from a June 2026 run.