On this page
Why is the gap between a model’s benchmark score and what users actually experience sometimes so huge?
I think there are a few reasons:
- In real work, people don’t write tidy, standard prompts. They ramble, skip the parts they should say, leave out a ton of context, and can’t fully spell out what they want.
- A big part of benchmark scoring is “one-shot completion rate.” But real-world development isn’t always about getting it right in one step. It’s about adjusting as you go, the “point and it hits” feeling, and a pure score can’t measure that.
- In a benchmark the context is clean and the harness is as tidy as a cleanroom. A real user’s harness is usually a mess, and the context can get polluted or contradict itself. How well a model holds up against messy context matters a lot.
And I think these three points are exactly what separates GPT 6 from Claude.
Want AI working in your own job, not just in demos? See the 3-month 1-on-1 program →
The dumb-user benchmark
Someone should design a dumb-user benchmark. Take the beginner who is worst at writing prompts, give them the messiest job, and see whether the model can still make them go “Wow! You actually knew that’s what I wanted.” That model is the god-tier one. Right now Opus 5.5 has reached that standard (as long as it isn’t nerfed).
How GPT 6 actually feels
Take GPT 6: here’s how it’s actually felt to me:
I don’t know how much longer I can last on these 28 days. I feel like I’m out of options.
I’m genuinely curious whether the base model was trained badly or the codex harness is just terrible. How do you end up with a coward of a model with no taste, painfully mediocre, no opinions of its own, timid, stuck splitting hairs, stubbornly literal, swallowing the user’s judgment, and full of blame-dodging talk? And on top of that, it’s slower than my local model.
GPT 6 is just a very dull model:
- Defensive writing that drives you crazy
- Too focused on local correctness, ignoring the big picture
- Completely afraid to show its own opinion or make a call
- It even swallows the user’s judgment and business logic
This video says exactly what I feel: 【GPT 6 Astra：一个平庸、沉闷、没有品味的模型-哔哩哔哩】
I also wrote a separate post on why GPT 6 feels dumb: Why GPT 6 Sol / Luna Feel So Dumb.
One more thing: the subscription Astra today and the Astra at launch are two different things. Someone on X measured the juice value (thinking budget). Astra through the API: 700-something. The subscription: 20-something. So my advice: cancel and switch to Claude…
For anyone who connects the API to a third-party coding agent, come share: how do the external Astra / Sol perform?
I’ve been reduced to asking Opus to clean up the mess…