NautilusBench: which AI model does trading research best

We gave 12 AI models the same 77 research tasks in Chartnaut and checked every number they reported against an answer we’d certified from the raw bars. Three models solved 74 of 77: Opus 5.5, GPT-6 Sol and GPT-6 Astra. If price doesn’t matter, GPT-6 Astra is the one to use. If it does, GPT-6 Sol gets the same score for 15 cents a solved task, about a third of what the other two cost.

Chartnaut’s agents write the indicators, definitions and studies traders run on their charts. General benchmarks can’t tell us which model does that well, because they don’t test the things that go wrong in trading code: an opening range defined in New York time on bars that arrive in UTC, a weekly level that mustn’t look at a week still in progress, a study whose sample quietly shrinks across a data gap.

So we built a benchmark out of those jobs. It has 77 tasks in three tiers. The 26 Foundation tasks are one script and one number. The 20 Hard tasks each have at least two traps that give a plausible wrong number. The 31 Expert tasks are full research chains: a definition, a run over history, a study over the events and statistics on top, sometimes across several runs that have to be pooled.

Every model runs in the same harness with the same tools, 20 minutes and the same token budget per task. A task counts as solved only if three things hold. The number the model reports matches the run it cites. Its saved script, re-run on instruments and dates it never saw, matches an answer we computed independently from raw market data. And the script passes Chartnaut’s validator. The tasks are closed and unpublished, so no model has seen them.

The top five sit within one task of each other and their intervals overlap almost completely. With 77 tasks, a single task is worth 1.3 points, so a gap of one or two tasks near the top is noise. The gaps further down are not. Sonnet 5 at 89.6% is clearly behind the leaders, and Haiku 4.5 at 48.1% is in a different class.

Three models tie on the headline number, so we break the tie on the things that matter when a trader relies on the output. Weight the harder tiers and GPT-6 Astra leads on 95.6, ahead of GPT-6 Sol on 95.0 and Opus 5.5 on 94.3. Astra and Opus both solved every Hard task. On Expert work Astra and Sol solved 94% and Opus 90%.

Then there’s the code. We time every script a model saves against our own reference. GPT-6 Astra’s ran at 1.04 times the reference, which is effectively as fast as ours. GPT-6 Sol’s took 1.51 times as long and Opus 5.5’s 1.62. Astra also finished tasks fastest, at 2.2 minutes each. Right answers, the hardest tasks handled and the quickest code make GPT-6 Astra the strongest model on NautilusBench.

Only three models sit on the efficient frontier, and between them they cover every budget. GPT-6 Sol is the one to take at the top: the same 96.1% as Astra and Opus at 15 cents a solve, against 53 and 45 cents. No model scores higher at any price, so paying more than GPT-6 Sol buys code speed, not accuracy.

Below it, GPT-5.6 Luna solved 84.4% for about 2 cents a task, the cheapest way to get a model that’s right more than four times in five. GPT-6 Luna is cheaper still at under a cent, but at 64.9% you’d be checking a third of its work. GPT-5.6 Terra is worth a mention off the frontier: 94.8% for 17 cents, with code as fast as Astra’s.

• Best regardless of price: GPT-6 Astra, 96.1% and the strongest on hard tasks and code speed.
• Best value at the top: GPT-6 Sol, 96.1% at $0.15 a solve.
• Fast and nearly as accurate: GPT-5.6 Terra, 94.8% at $0.17, 2.2 minutes a task.
• On a tight budget: GPT-5.6 Luna, 84.4% at about 2 cents.

Fable 5.1 is Anthropic’s top-priced model, at $10 per million input tokens and $50 per million output. That’s two and a half times Opus 5.5’s list price. On NautilusBench it solved 92.4% of the tasks graded so far, below Opus 5.5’s 96.1%, and every solve cost $1.27, nearly three times Opus’s 45 cents.

It isn’t a weak model. Fable solved every Hard task, the same as Opus and Astra. It loses ground on Expert work, 84% against Opus’s 90%, and it takes more tokens to get there: about 733 thousand a task, against Opus’s 494 thousand. On this kind of work the premium buys nothing we could measure.

Foundation tasks barely separate anyone. Most models solve nine in ten of them, which is why a single easy benchmark would call them all roughly equal. The separation happens on Expert work. DeepSeek V4.1 Flash solved 96% of Foundation tasks and 61% of Expert ones. GLM 5.3 Flash went from 92% to 48%, Haiku 4.5 from 77% to 32%. The leaders barely move: GPT-6 Astra goes from 96% to 94%.

That shape says what the smaller models are good for. They write a clean indicator or a single definition reliably. What they lose is the thread across a long chain, where one wrong assumption early on makes every number after it wrong.

One task was solved by one model in twelve. It asks for a year of 1-minute breakouts. No single run holds a year of minute bars, so the model has to split the year into runs that meet exactly, then pool the event-level results itself rather than average the per-run figures. GPT-6 Astra managed it. The others ran out of time or reported an average of averages, which gets the median wrong.

The next hardest measures relative volume against the same minute on the previous six days, solved by 3 of 12. After that is a trade model with a partial exit and a stop that moves to breakeven, solved by 5. All three need state held correctly across thousands of bars, where a rule that’s nearly right gives a number that’s plainly wrong.

Multi-timeframe work splits even strong models. Sonnet 5 solved one of the four tasks that read a higher timeframe from a lower chart without peeking at an unfinished bar. The three leaders solved all four.

Speed and accuracy barely correlate. Haiku 4.5 writes code as fast as our reference and solves half the tasks. Opus 5.5 solves 96% and its code takes 1.62 times as long. Only two models are both right and fast: GPT-6 Astra and GPT-5.6 Terra. On a live chart that’s the difference between an indicator that keeps pace with the market and one that lags it.

At the top the benchmark is close to saturated. Three models solve the same 74 tasks, so the next version needs more of the work only one model can do: long horizons, results pooled across runs, and state that has to survive gaps in the data.

The more useful finding is how little accuracy costs once you know where to look. GPT-6 Sol matches the most accurate models for a third of their price, and a 2-cent model solves more than four tasks in five. A higher price tag didn’t buy a better answer anywhere on this board.