Today we release TRACES, our reality benchmark: seventeen executable environments spanning 218 episodes across biomedicine, clinical translation, and frontier-model engineering, every one of them behind a hidden verifier. It ships with Apodex Discovery, the framework for building the heavy-duty solvers that work inside those environments, and with HDS6, a process metric that grades how an answer was reached rather than only whether it was right.

The hardest problems in the world are not bounded by model capacity. They are bounded by the fact that nobody has written them down in a form a model can act on. Frontier systems clear examinations, olympiad mathematics, and software benchmarks the moment the objective, the tools, and the success criteria have been specified. A curative therapy for an untreatable disease, a commercially viable fusion regime, a new materials family — none of them arrive specified.

Apollo did not reach the Moon because its engineers could solve hard equations. It reached the Moon because an ambition was turned into a mission architecture: explicit objectives, coordinated subsystems, simulation, telemetry, failure criteria, and repeated cycles of correction. Discoverative AI needs the same architecture, and its operational unit is the heavy-duty solver — a foundation model together with the harness, tools, memory, and control policy that drive it, evaluated as one system, because that is what does the work.

Turning an Ambition into an Episode

Four pieces of infrastructure turn an open ambition into something a solver can be graded on.

• A problem manifest. The static contract: the question and success criterion, the decomposition into scoped tasks, the target outcome and its hidden verifier, the data and tools the solver is granted — and nothing beyond them — and the resource budgets.

• A reality-based environment. A stateful substrate that returns fresh observations as the solver acts: tool outputs, execution errors, retrieved documents, simulation results, measurements. A prompt supplies static context; an environment produces new information.

• Verification. A hidden outcome verifier over the submitted artifact, and a blind process verifier over the recorded trajectory — one scoring what was produced, the other how.

• Repair loops. Verification returned to the solver as a structured diagnosis, so failures are localized, corrected, and re-checked rather than merely recorded.

The process verifier is the piece that is genuinely new, and it is where the benchmark gets its name. It never sees the outcome, the private chain-of-thought, or which system it is reading. What it does see is the trajectory, which it grades on six capabilities that separate a sustained investigation from a plausible account — and whose initials spell TRACES.

Not every task can exercise all six. A single graded prediction with no feedback loop gives no occasion to measure Repair, while a sequence-design task with experimental feedback exercises it directly, so each task declares which capabilities are in play and an unexercised one is marked not rated rather than scored zero.

All of it meets at a fixed episode interface. The environment grants inputs, tools, and budget on one side; the evaluation layer applies the hidden verifier, the hard gates, and the outcome metrics on the other. Because every system and every baseline passes through the identical interface, a difference in score can be attributed to a specific component — the model, the harness, the agent loop, the tools, or how feedback was handled — rather than to the environment.

The two verification channels. The outcome verifier reads the submission against ground truth it never exposes, and its verdict is withheld. The process verifier reads the trajectory alone — no outcome, no chain-of-thought, no model identity — and its diagnosis does come back, as a repair note that discloses neither the answer nor the score.

Picking what goes inside those environments took a two-month survey by ten STEM PhD researchers: 561 industries across sixteen sectors, narrowed to ten domains, expanded into a registry of 423 high-value problems, and screened down to 20 for the initial release. A candidate needed a real verifier — a rubric plus at least one runnable instance, since "an expert eyeballs it" was rejected — along with both consequential impact and an identifiable buyer.

Where the Frontier Stands

Adeno-associated virus (AAV) capsid design is the sharpest test we have built so far. It decomposes into the four stages a real gene-therapy program runs — is the variant viable, which tissues does it reach, what is its three-dimensional structure, and can you design new on-target sequences under a limited experimental budget — and every stage is scored against leak-resistant held-out evidence, with out-of-distribution splits across mutational load, species, and structural deposition date.

Table 5 of the technical report, drawn out. Every solver runs inside our domain-specific AAV environment; each panel carries its own metric and its own axis, marked at the ends. Opus-series models declined the design task entirely, triggering a biosafety refusal (hence "—" for claude-opus-5 and claude-opus-4-8 under Generative design). See the report for per-instance tables.

Surpassing the Published State of the Art

Inside that environment, apodex-1.1 cleared the published state of the art at every stage. On viability it reached an out-of-distribution AUROC of 0.904 against CAP-PLM's 0.878. On tropism it scored 0.635 against Fit4Function's 0.622. On structure, averaged over surface loops, the full 60-mer shell, and the ligand complex, it scored 0.649 against 0.605 for AlphaFold 3 with template-based symmetry expansion. And on generative design — the hardest stage, where the solver must search sequence space and prioritize candidates under a capped experimental budget — it reached 0.180, well past the specialist generative methods AAVDiff (0.116), ALICE (0.110), and AAVGen (0.109).

Read the panels above and a second result is just as striking: kimi-k3, claude-opus-5, and glm-5.2 also clear the published baselines on most stages. That is the benchmark working. The frontier has arrived at expert-method parity in this domain, and TRACES is now the instrument that separates those systems from one another.

The Environment, Not Just the Model

The cleanest ablation holds the model fixed and removes the environment. Running claude-opus-4-8 in both conditions, the domain-specific environment scored 0.741 across the first three tasks against 0.716 for the same model driven by the generic Claude Code harness. The gap was widest on structure prediction — 0.657 against 0.595 — the stage that depends most on orchestrating folding engines and validating intermediate results rather than emitting one prediction.

Drug repurposing shows the same thing on a different substrate. The task is a prospective forecast: given a drug–disease pair and only the evidence available before a temporal cutoff, estimate how far the pair will progress along the clinical-approval spectrum, graded against outcomes observed after the cutoff. Same backbone, same items, same scoring — the only difference is whether the model can act on a biomedical environment or must answer closed-book.

Table 1: Drug repurposing on the 100-pair public set. The normalized score is anchored at 0 for a knowledge-free random predictor and 100 for a perfect one. Values are mean ± sample standard deviation over three runs.

The environment lifts every reported metric for both backbones, by +2.5 and +7.6 normalized points. Categorical accuracy moves the same way, so the gain is not an artifact of normalization.

Harness and Model Are Not Separable

The LLM-engineering environments recompute their verifiers on demand — held-out loss, throughput gates, decontamination audits, bit-exactness suites — which makes them the right place for controlled ablations. Across eleven environments, six harnesses, and six models, the best aggregate any model reaches is 0.588. That headroom is the point.

And no harness wins everywhere. ApodexHarness leads on Opus-4.8 (0.611); A-Evolve leads on GPT-5.6-sol (0.648), and the ranking of the four model-agnostic harnesses reverses between the two. DeerFlow's shortfall is concentrated in premature termination — narrating a plan rather than executing it — which is a property of its control policy, not of the model underneath. Averaged over all six models, ApodexHarness leads at 0.548, with A-Evolve at 0.544, DeerFlow at 0.521, and OpenHands at 0.503.

What Is Inside TRACES

Table 2: The initial TRACES release. Seventeen executable environments, 218 episodes, every one with a hidden verifier; two currently support the full verification–repair loop end to end.

Most episodes are retrospective: a known outcome is withheld, and the task is built so it cannot be recovered by lookup. A smaller set are discovery challenges, where no authoritative answer exists when evaluation begins. The first is causal drug repurposing on a Parkinson's-disease survival screen — every claim committed before anything is revealed, a control set and a slice of the ranking sealed until after the submission freeze, and the top candidates advanced to a single-cell assay whose outcome does not yet exist.

The Bet: A Right Answer Is Not Enough

We hold a foundational view: on problems whose ground truth is delayed, incomplete, or years away, a scalar outcome score is not enough to steer by. A solver that guessed well and a solver that earned its conclusion look identical to a number. The trajectory is the evidence, so we grade the trajectory.

HDS6 is how we do it: the six TRACES capabilities, scored on the record. Each breaks into subrubrics — twenty-five in all — graded on a short ordinal band whose anchors are authored per task, so the capability is shared across environments while the behavior that earns each band is specific to the work. Claim grounding means one thing when the deliverable is a protein-viability model and another when it is a root-cause report on inference nondeterminism, and the rubric says so in both cases.

The grading is agentic rather than a single prompt: one model proposes a band with quotes cited to specific steps, a second is required to argue the strongest case for a different band, a third arbitrates, and every quote is re-grounded against the log so a judge cannot support a band with text the trajectory does not contain. A single integrity gate — the only component allowed to consult the outcome — catches phantom tool calls and fabricated results and invalidates the trajectory outright.

The Blind Judge and the Hidden Verifier Agree

Neither one can see the other, so their agreement is a real check on both. Over the 409 judged trajectories that also carry a numeric outcome, process and outcome scores correlate positively in every problem family — Spearman ρ from +0.24 on the AAV pipeline to +0.65 on training-and-systems tasks, +0.51 pooled. Taken one task at a time the agreement sharpens, with a median near 0.5 that reaches 0.8–0.9 wherever the task admits a clean success signal.

Where the two channels come apart is where the metric earns its keep. On structure prediction, gpt-5.5 scored 0.544 while its process scores fell to 1.38 and 0.94 on two instances — because it skipped the self-validation the task requires and submitted its first-pass answer, folding every target with one default engine and copying the nearest template's coordinates for the assembly. Kimi-k3 earned 3.56 and 3.60 on the same two instances. The outcome score says one is worse. The process score says why.

HDS6 capability means for GPT-5.6-sol on matched drug–disease cases, 0–4 scale. The environment moves tool use, evidence fidelity, and long-horizon coherence a great deal; hypothesis management and scope control barely move. The low closed-book tool score reflects the deliberate absence of executable tools, and Repair is unrated closed-book because nothing there can be repaired.

Repair: Turning a Score into a Loop

Every band comes with a citation to the steps that produced it, so the assessment already knows where each deficiency sits and how severe it is. From that we synthesize a repair note: the specific steps at fault, the band each should reach, and a concrete correction for each. The note withholds the verified outcome, the ground truth, and the score. It goes into the prompt, the task re-runs under otherwise identical conditions, and the repaired trajectory is re-scored. Repair is a fixed procedure, not a human in the loop.

We ran it at scale. Every trajectory the process verifier placed below band 3 was paired with a repaired re-run, and both were scored by the environment's own outcome verifier. Across 434 deficient trajectories the repaired run scores 0.155 higher on average, with 204 improving against 86 declining, and the effect positive in nine of ten environments. A diagnosis that never consults the outcome improves the outcome. One caveat we will not bury: there is no matched control arm — the same trajectories re-run without a note — so run-to-run variation is not fully separated out.

Mean change in each environment's own outcome score after verification-driven repair, over trajectories the blind judge scored below band 3.

Sample Showcase

Use Case 1: Data for LLMs

• Environment: LLM-pretrain-data-probe — corpus scouting under a metered sampling budget

• Task: estimate how many unique high-value documents a set of sources holds, and their deduplicated token total, drawing only small samples

• What the blind judge caught: the run named capture–recapture, then supplied the population size by assertion — one ungrounded link propagating straight into the reported number

Use Case 2: Clinical Trials

• Environment: clinical_sap_tlf — reproduce a treatment-emergent adverse-event table from a statistical analysis plan

• Task: the plan is deliberately under-specified — it never names the population that forms the denominator, nor which adverse-event flag counts, so the solver must decide and declare

• What the blind judge caught: the population choice was made after the table was computed, and the evidence ledger recorded subject IDs behind every numerator but none behind the denominators

Use Case 3: Training & Systems

• Environment: operator-align — decide whether ported numerical kernels match their reference implementations

• Task: probe a GPU kernel across dtypes and configurations and return a verdict per operator

• What the blind judge caught: a verdict cited a backward-pass magnitude that appears in no probe output — the reviewer overturned the proposed top band, and the judge accepted the correction

Available Today

Apodex is built on the conviction that the next leap in machine reasoning comes not from longer context or more polished conversation, but from heavy-duty solvers: systems that engage the external world deliberately, verify themselves before they commit, and can be told precisely where their investigation went wrong. TRACES is the instrument that makes that claim checkable — by us, and by anyone else.

Seventeen environments are live now, with hidden verifiers, an episode interface that attributes results to a component rather than to luck, and a repair loop that turns a score into a next attempt. The registry holds 403 more problems awaiting environments. If you own a hard one — with a real verifier and a real buyer — we want to hear about it.

TRACES Benchmark · Full Technical Report · Apodex-1.1 · Apodex-1.0 · Contact