How am I supposed to read this AI benchmark chart?
A walkthrough of overlapping dots, effort settings, cost frontiers and percentages in recent AI coding benchmarks.
Loading content...
A walkthrough of overlapping dots, effort settings, cost frontiers and percentages in recent AI coding benchmarks.
Some AI benchmark charts are incredibly difficult to understand and intuit. I work through the axes, effort settings and measurements in recent examples, with diagrams and readable values.
Retrospective archive placement: October 26, 2026. First published October 27. Cursor table values and the AA original graph were checked October 27; the AA launch snapshot remains dated October 12.
I want to understand these charts. I also think some of them are incredibly difficult to understand and intuit.
There are model names, percentages, dollar amounts, colored dots, and lines crossing other lines. A new model comes out, someone posts a graph, and the caption makes it sound obvious what I should take away from it.
It often isn't obvious to me.
This October 26 chart from Cursor is a good example. It compares coding benchmark scores and cost across several models and effort settings. There is useful information in it, but I have to work through several different questions before I can use that information.
Let's do that, rather than start with a simple bar chart that doesn't have the problems I'm talking about.
Newest dated examples first, then teaching diagrams. Nothing advances automatically.
Original example: · Checked 2026-10-27
CursorBench 4.0
October 26 original post (new tab) · Current table and methodology (new tab)
Average cost per task (USD) · linear · cheaper to the right
DeepSeek V4.1 Flash 1017Opus 5.5
Higher scores go up. The dollar scale runs from $18 on the left to $0 on the right. Better score and lower cost point toward the upper right. The score range is 20–60%, not 0–100%.
| Model / option | Setting | Score | USD / task |
|---|---|---|---|
| DeepSeek V4.1 Flash 1017 | Low | 44.9% | $0.60 |
| DeepSeek V4.1 Flash 1017 | High | 45.6% | $0.77 |
| DeepSeek V4.1 Flash 1017 | Medium | 46.3% | $0.80 |
| DeepSeek V4.1 Flash 1017 | Max | 47.7% | $0.98 |
| Opus 5.5 | Low | 43.7% | $1.17 |
| Opus 5.5 | Medium | 52.5% | $2.91 |
| Opus 5.5 | High | 56% | $3.97 |
| Opus 5.5 | Extra High | 56% | $6.98 |
| Opus 5.5 | Max | 57.8% | $13.43 |
In the Cursor chart, the vertical axis is a benchmark score. Higher is better.
The horizontal axis is average cost per task. The dollar amounts decrease from left to right: $18, $15, $12, and eventually $0. Cheaper is farther right.
So the attractive corner is the upper right.
That takes a moment. Usually, moving right means getting more of whatever the horizontal axis measures. Here it means spending less. If I miss that, I can read the entire chart backwards.
Reversing an axis can be a reasonable choice. It puts both desirable directions together. But the direction needs to be clear before I look at the model names.
My first questions are:
Even the units need attention. Dollars per task, dollars per million tokens, seconds per response, and tokens per second answer different questions. I can't substitute one for another just because all of them appear in discussions about which model is cheaper or faster.
A dot needs more than a model name. It also needs a setting and a test.
The highlighted DeepSeek V4.1 Flash 1017 Max result is 47.7% on CursorBench 4.0 at $0.98 average cost per task. The same live table lists Opus 5.5 Max at 57.8% and $13.43. These are the values checked on October 27, rather than permanent descriptions of either model. CursorBench results
For this comparison, one option buys a higher measured score at a much higher measured cost. Which would I choose? That depends on what I need, what I can spend, and whether this test resembles my work.
The chart can't choose those priorities for me.
The lines make this harder because a line often suggests time. It is easy to see an upward or downward slope and think something is improving or getting worse.
Here the connected points represent different settings of a model. They aren't a history of that model's performance. And a setting called “high” on one model isn't a standardized amount of computation shared by every other model.
In the walkthrough, I can isolate one result, add a second, and then reveal the selected effort settings. The full data table keeps each setting attached to its values.
The Cursor post emphasizes a cost-frontier claim. That phrase deserves an explanation before it becomes the headline.
Imagine three hypothetical options:
| Option | Average cost per task | Score |
|---|---|---|
| A | $1 | 50 |
| B | $2 | 50 |
| C | $3 | 60 |
B costs more than A without improving the score. If these two measurements were all I cared about, I wouldn't pick B.
A and C present a real choice. A is cheaper. C scores higher.
The Pareto frontier is the set of options that aren't beaten by another option on both measurements: no more expensive and at least as good, with a strict improvement in one of them. In this small example, A and C are on it. B isn't.
That doesn't make A and C equally useful. It tells me where the trade-offs are. I still have to decide whether the extra score is worth the extra money.
It also depends on which options were included. Add a fourth option that costs less and scores higher, and the frontier changes.
A line connecting one model's effort settings and a line identifying the frontier can look similar. They have different meanings. I need the legend to tell me which one I'm seeing.
The Cursor chart reaches roughly $18 while the highlighted comparison is under $1. On that linear scale, sub-dollar differences occupy a small strip near the right edge. Labels and lines fight for the same space.
The detail step above opens a sub-dollar panel while retaining the full view. It leaves the expensive Opus Max point outside the detail window, so the caption names that omission.
Another possibility is a logarithmic cost axis. That brings its own reading rule: equal distances mean equal ratios, rather than equal dollar differences.
For example, hypothetical ticks at $0.10, $1, and $10 are tenfold steps. The jump from ten cents to a dollar occupies the same space as the jump from a dollar to ten dollars, even though the dollar increases are very different.
The Cursor example uses a reversed linear axis, not a logarithmic one. I don't want to see a dense cost chart and assume every author made the same choice.
The vertical range changes the visual comparison too. This image displays scores from 20% to 60%, rather than the entire 0–100% range. That makes differences easier to see. It also means the visible height isn't a proportion of the full percentage scale.
A dot that looks twice as high above the bottom of the plot doesn't have twice the score. Read its value. For bars, a nonzero baseline needs particular care because the length of the bar is part of what communicates magnitude.
This is where I need to open the methodology, not keep staring at the picture.
Artificial Analysis's October 12 FrontierCode launch article provides a useful example. Its leading launch result, Claude Opus 5.5 at max effort with fallback, had a 52% score and a 58% pass rate. Those numbers describe different things. Passing means clearing the blocking criteria; the score also reflects non-blocking criteria. The launch article preserves its October 12 results and points separately to a live leaderboard.
So I shouldn't turn every number followed by a percent sign into “this model completes this percentage of my work.” Even within one benchmark, the score and pass rate can differ.
Before comparing two percentages, I need to know their denominators and grading rules. Were these tasks passed? Rubric points earned? Human preferences? An average across several tests?
And if an average includes very different tasks, I want to see the parts.
The Artificial Analysis Coding Agent Index currently combines three benchmark components with equal weight. That gives a convenient summary. It doesn't tell me that each component resembles my own workload equally well. Its methodology identifies the components and their scoring rules.
For a coding decision, I would want to know whether an apparent overall advantage came from repository editing, terminal work, or answering questions about a codebase. A summary is a starting point for that investigation.
A model does not work alone in an agent benchmark. Something supplies its tools, manages the conversation, gives it a workspace, and decides when it has run out of attempts or time. That surrounding software is often called the harness.
Change those conditions and I may be comparing different systems, even if the model names look familiar.
The current FrontierCode evaluation page describes Artificial Analysis's implementation, including its Stirrup harness and repeated attempts. That is useful context when comparing it with results produced elsewhere. Sharing a benchmark name doesn't automatically establish identical execution conditions.
I would look for:
Those details belong next to the chart, or one obvious click away.
Cursor calculates its average task cost from published token prices applied to the tokens used in the evaluation, including cache categories. It isn't the price of a Cursor subscription. It also doesn't promise that my next task will cost that amount. Cost definition
A low token price can be offset by using more tokens. A task that uses fewer tokens can still take longer. A fast first response can still leave me with substantial checking and cleanup.
If my concern is waiting, I need a time measurement. If my concern is spending, I need a cost measurement. If my concern is finishing a job, I need to consider the checking and correction that happens afterward.
The graph may measure only one or two of those things. I shouldn't silently supply the others.
Cursor explicitly warns that small score differences may not be statistically meaningful. Its posted image doesn't show uncertainty bars. Results and caveat
I can describe which displayed number is larger. I need more evidence to say that a tiny gap represents a dependable advantage.
Error bars, repeated runs, and task counts can help, but I also need to know what the uncertainty calculation covers. Variation between repeated attempts is one question. Whether the chosen tasks represent my work is another.
Absence of an uncertainty interval doesn't prove the results are unreliable. It means the image hasn't answered that question for me.
The annotated examples above are grouped by chart type. Within each group, dated examples appear newest first; undated teaching diagrams follow them.
Choose a graph type, then use Previous example and Next example. Each example has numbered walkthrough steps, source and check dates, and buttons for moving through the explanation. Exact plotted values are available in a readable table. Nothing advances automatically.
The original AA graph uses an increasing logarithmic cost axis, so its attractive corner is upper left. Cursor's graph uses a decreasing linear cost axis, so its attractive corner is upper right. The walkthroughs keep those rules separate. The AA cost redraw uses rounded values stated in its launch text, rather than claiming pixel-level precision from the original chart.
A newer chart can be added without erasing an older example's source, benchmark version or scoring rules.
Something like this:
“This setting scored higher on this version of this test. It also cost more under the evaluator's pricing calculation. The cheaper setting may still be enough for me. I need to check the task mix and the uncertainty before treating the difference as decisive.”
That is less exciting than declaring a winner. It is much more useful when I'm deciding what to use.
I don't need every chart to be simple. I do need to be able to work out what it says, what it leaves out, and whether it helps answer my question.