A task you can't fake

Start with one of the tasks:

Task 03 · 23 ground-truth dimensions · wrong claims score below zero

As of January 1, 2026, how many flight tests has SpaceX's Starship completed in total? Please list them one by one, with the key progress and the failure points of each test.

On its face this is a counting question, and the count is almost certainly in every recent model's training data. But the scorer checks 23 ground-truth dimensions: the progress and failure points of all flights have to line up, and a wrong claim scores below zero. Reciting the headline number is easy; listing every flight correctly without saying anything false takes real retrieval and real checking.

Over the past year, deep research has become a standard feature of agent products, but evaluation has not kept up: scores keep multiplying while they mean less and less. We put the effort into the instrument itself: tasks designed to resist memory-only solving, and scores that can be independently reproduced under the same evaluation configuration.

Today we are open-sourcing FrontierSearchBench: 41 verifiable deep-search tasks, each requiring a long chain of live web retrieval and scored by its own program. The task set and all the scorers are at https://github.com/ApodexAI/FrontierAgent/tree/main/benchmarks/frontier_search_bench.

Three ways deep-search evals fail

Existing evaluations break in three places.

Contamination. Once a static QA set is published it flows into the next generation's training data. From then on you are measuring memory, not retrieval.

Judge drift. Report-style tasks get scored by a rubric plus an LLM judge. Swap the judge or nudge the prompt and the score moves: same answer, two grades on two days. What doesn't reproduce can't be compared or tracked.

Not enough depth. BrowseComp asks agents to find one needle, even when finding it takes many hops. Real research often asks for the whole haystack: an enumerated set, ordered list, or numeric summary whose components must all be collected, reconciled, and verified.

Whether a deep-search evaluation deserves trust comes down to three questions: does solving the task require live, multi-round retrieval? Will the answer still be correct next year? Would a different judge give the same score? FrontierSearchBench is built so the answer to all three is yes. Deep search is a chain: decompose, search, aggregate, verify. The first two are commodities by now; the results below show the separation happens in the last two.

How the benchmark is built

Four design principles:

• A structured deliverable per task: scoring checks a set of ground-truth dimensions, not one answer string; components are gathered from many sources and reconciled.

• Answers don't drift: 23 tasks carry an explicit temporal anchor ("As of January 1, 2026"), and 18 are closed-world ("all INES level-4-or-above nuclear accidents, 1945-2026" is a closed set). Today or next year, the ground truth is the same.

• Tasks come from the real world: the three Singapore condominiums with the highest 2025 resale returns among projects that obtained TOP in 2019, the 2020-2025 admission cutoffs for computer-science programs at 985 universities in Henan province, Nobel laureates whose prize-winning paper was first rejected by a journal. Investors, analysts, and journalists actually ask these; none was constructed backward from an answer key.

• Contamination resistance is structural: components are scattered across many live web pages, and memory alone cannot assemble the deliverable; tasks, annotations, and scorers were finalized before, and independently of, any of the runs we evaluated.

Scoring runs in four stages: the delivered report is extracted into structured claims (the one stage run by judge models, pinned to fixed versions), the claims are aligned to the task's ground-truth dimensions, a task-specific rubric scores each dimension, and the task total lands in [−1, 1] with wrong assertions pushed below zero. The benchmark score is the mean over all tasks.

The answers do not live in the task file (queries/verifiable.json contains only the questions); they are encoded in each task's own scorer (task 3, excerpted):

"""Query 03 — SpaceX Starship IFT auto-scorer.

Scoring:
  A_flight_count       ±1   (accept ∈ {10, 11}; missing → -1)
  Per IFT-N (N=1..11):
    progress sub-dim   ±1   (any wrong claim → -1; else any correct → +1)
    failures sub-dim   ±1   (same rule)
  Total ranges from -23 to +23.
"""

PER_IFT_GT = {
    "IFT_1": {
        "date": "2023-04-20",
        "progress": [...],   # canonical claims + keyword matchers
        "failures": [...],
    },
    ...
}

"""Query 03 — SpaceX Starship IFT auto-scorer.

Scoring:
  A_flight_count       ±1   (accept ∈ {10, 11}; missing → -1)
  Per IFT-N (N=1..11):
    progress sub-dim   ±1   (any wrong claim → -1; else any correct → +1)
    failures sub-dim   ±1   (same rule)
  Total ranges from -23 to +23.
"""

PER_IFT_GT = {
    "IFT_1": {
        "date": "2023-04-20",
        "progress": [...],   # canonical claims + keyword matchers
        "failures": [...],
    },
    ...
}

"""Query 03 — SpaceX Starship IFT auto-scorer.

Scoring:
  A_flight_count       ±1   (accept ∈ {10, 11}; missing → -1)
  Per IFT-N (N=1..11):
    progress sub-dim   ±1   (any wrong claim → -1; else any correct → +1)
    failures sub-dim   ±1   (same rule)
  Total ranges from -23 to +23.
"""

PER_IFT_GT = {
    "IFT_1": {
        "date": "2023-04-20",
        "progress": [...],   # canonical claims + keyword matchers
        "failures": [...],
    },
    ...
}

Rerunning the model is not expected to produce an identical score: the web changes and retrieval is stochastic, as with any live-web benchmark. The evaluation itself, however, is reproducible by construction. Ground truth, rubrics, extraction prompts, and judge model versions are all pinned, so any submitted answer can be rescored under exactly the same evaluation configuration.

Anatomy of one task

Back to the opening task. The scorer defines 23 dimensions: one for the flight count, accepting {10, 11} (whether the suborbital hops count is a defensible disagreement), with a missing answer scoring −1; each of the eleven flights then gets a progress and a failures dimension: correct claims earn the point, silence earns nothing, wrong claims cost one. The total spans [−23, +23], normalized to [−1, 1].

The Agent Team run (Apodex-1.1): the orchestrator splits the task across four subagents (official sources, trade press, news corroboration, recent flights), which together issue 210 retrieval actions (66 searches, 144 fetches); after the merge, a final verifier re-checks the assembled answer against primary sources, confirms the skeleton (all eleven flights, with dates, serial numbers, and outcomes correct), catches 8 detail errors (Flight 1's engine event belongs at T−6 seconds in the ignition sequence, not after liftoff), and fixes each one before submitting. Score: 0.61.

Same model under single-agent ReAct: the headline number is just as correct and the 9.7 KB report is respectable, but it scores 0.30. No verification step; several detail claims tripped the wrong-assertion penalty.

Figure. Two measured runs of Apodex-1.1 on task 3 (the Starship flight-test history), per-task score in [−1, 1]: same model, same scorer, different harness. Wrong assertions score below zero; the external verification pass is the difference between 0.30 and 0.61.

Two conclusions: the discriminative power of a deep-search evaluation is not in the headline number but in the long tail of the structured deliverable; and external verification is worth points, with a 0.31 absolute gain.

Results

The API baselines all run the ReAct harness; each Apodex row names its own. The two Apodex-1.1 rows are the same model under two harnesses.

All rows were measured by us under one set of conditions: the same judge panel, a 262,144-token context window, and a 32,768-token completion budget. Per-task scores lie in [−1, 1]; the first three columns bin tasks by score sign, and Avg. is the mean scaled by 100. Bold marks the best value in each column.

Apodex-1.1 with ReAct is competitive with the open-weight systems, while Apodex-1.1 (Agent Team) surpasses the strongest proprietary reference shown, achieving the best overall result and ranking first in both average score (69.1) and positive-task coverage (87.8%, 36 of 41 tasks).

The gap between the two Apodex-1.1 runs points to what FrontierSearchBench is actually measuring. Both harnesses can retrieve enough information to produce a plausible answer; the separation comes from what happens after retrieval. The Agent Team run goes back to primary sources, catches detail-level errors, and turns a 0.30 into a 0.61. That is exactly the behavior the benchmark is designed to reward: not just finding evidence, but reconciling and verifying it before making a claim. The same standard applies to the benchmark itself. A score should not depend on an evaluator’s impression: the ground truth, rubrics, extraction pipeline, and judge versions are pinned, so submitted answers can be rescored and disputed under the same configuration.

Every frontier model we tested still comes back with zero or negative scores on some tasks: real questions on the open web are still the frontier. The instrument is open source: come measure.

https://github.com/ApodexAI/FrontierAgent/tree/main/benchmarks/frontier_search_bench