We invite researchers, model developers, and agent builders to submit their models and solvers to TRACES and take on high-value problems drawn from science, medicine, engineering, industry, and beyond.
Bring a method to problems that have not been solved yet.
See how a frontier model performs when there is no answer key.
Put the whole harness — loop, tools, memory — to work.
| Environments | Your system runs inside our executable environments — the same stateful sandboxes, the same action interface, the same budgets for every submission — so a difference in outcome is attributable to the solver. |
| Data & tools | It works with domain-specific data and tools curated for each problem: the sources, instruments and evidence a human expert would actually reach for on the way to a solution. |
| Evaluation | We report how it performs not only on the final answer, but across the entire journey to a solution — the trajectory it took, and how well it navigated the unknown. |
A single accuracy figure cannot tell you whether a system is reliable, whether it got lucky, or whether it is affordable to run.
| Metric | What it measures |
|---|---|
| Outcome score | Did it solve the problem? Graded by a hidden verifier — not an answer key. |
| TRACES score | Did it earn the answer? Six capabilities — Tools, Repair, Alternatives, Coherence, Evidence, Scope — scored blind to the outcome. |
| Cost | Tokens, tool spend and GPU minutes per episode, in dollars. |
Pick whichever fits how your product is built. The first is by far the most common and takes about a day of your team's time; the third gives you the most representative score and takes real engineering.
| Route | What you give us | Best for |
|---|---|---|
| Route A — API key | Endpoint URL, key, exact model name, rate limits, price per token | API model providers |
| Route B — ckpt | Weights, tokenizer, chat template, serving config | Unreleased or on-prem models |
| Route C — harness | A Docker image meeting our sandbox contract, plus a passing self-test | Agent products and labs with in-house scaffolds |
All submissions are governed by a written agreement with Apodex US, Inc., put in place before any materials are transferred, which supersedes the site terms for anything you provide.
* Route C runs two arms: Arm 1 — your model in our reference harness (the comparable, leaderboard-ready number); Arm 2 opt-in — your model in your own harness (shows what your scaffolding adds). If your harness can't run in time, Arm 1 still produces a complete profile.
No. An API key is enough and is the most common route. Weights are only for models you can't or won't expose as an endpoint.
Yes — that's Route C, and for agent products it's the number that matters. It requires a Docker image that runs in an offline sandbox, so budget engineering time, or ask us to do the integration. We always also run your model in our reference harness so your result stays comparable to everyone else's.
No. Ground truth and the verifier live on the host, outside the machine your system operates, and every observation your system receives is screened for leaked hidden fields before it is sent. In most environments the score itself is never shown during the episode.
Both depend on the domains, the number of episodes and your rate limits, so we quote them together once we know the scope — never before. Worth knowing in advance: the schedule is usually driven by your endpoint's quota rather than by our compute.
You get the environment descriptions, the action interface, the metric definitions and a practice environment — everything needed to prepare. The specific instances and the verifiers stay hidden, which is the whole point.
If the cause is ours — infrastructure, a broken environment, an upstream outage — we re-run it at no charge to you and disclose it in the report. If the cause is your system running out of budget, timing out, crashing or never submitting, that is a result and it counts.
“The future of AI benchmarks is not an exam. It is the world itself.”