If you’ve read the posts that arrived in the last week or two you might come away thinking “frontier models can now drive robots in real time”. The results are impressive. But that’s not quite exactly what’s happening. While some scenarios do try this, the results for that are not great. In the more successful scenarios the frontier model is building the systems that drive the robots, teaching them and checking their work - and then it’s stepping out of the real-time loop as quickly as it can. If you look a little deeper into each post then you’ll see the results point very clearly at where this likely takes us.
A - R - G - O - N - source GoogleThe important insights are packed away inside these demos: how long each call takes, what it costs, and how much the model has to remember to stay competent.
And as often happens when we pass another capability threshold, three very related posts all landed within a few days of each other. Google’s Gemini 4 Argon for Robotics post. NVIDIA’s Astranomy study of GPT-6 Astra as a robot policy. And Artificial Analysis’s new AA-3D benchmark for building interactive 3D scenes. If you read them together then they describe a single “bigger picture”.
The bigger picture
Understanding is expensive. But once you have that understanding, then questions are cheap.
The key is to only build the understanding once,
then make everything else a question.
That’s the argument from my earlier post A Unified Future For Computer Vision, 3D Generation & World Models, and this week put a tangible price tag on it. This picture has three core parts.
A maintained picture of the world
What exists, where it is, what’s happening now and what should happen next. It lives outside any single model, and it persists between questions.Two interfaces onto that picture
Perception reads the world into it. Generation writes new worlds out of it. Same shape, opposite direction.A loop that runs it
Small, fast, cheap components do the always-on work - checking what they see against what they expect, acting, correcting.
The best place for the frontier model to sit is above that loop: it builds the system, teaches the smaller parts, and judges the hard cases they escalate. It’s only called when the loop is unsure - not for every frame.
Each of the related posts I mentioned above fills in one part of this, and each one also confirms that the frontier model belongs above the loop, not inside it.
Argon: the frontier as harness author
Google’s post is a showcase. A Spot quadruped learns to step on a bin pedal after practising in a simulation it built itself. An arm rights a mug and puts a napkin in it with no task-specific training. A world model is steered to spell a word with scattered tiles. Egocentric human video becomes a dexterous humanoid simulation. A pen-and-paper sketch becomes a physics-verified throw. It also leads the embodied-reasoning benchmark at 87%.
Now read the prompts rather than watch the videos.
The arm demo tells the model to pick its own call budget up front - two to four calls for a coarse run, four to seven with a hover-and-verify step, eight to twenty for fine visual servoing. The actual motion is executed by code primitives: grasp, place-until-contact, move-until-contact. The fast loop is inverse kinematics and a controller. The model is only consulted a handful of times per task.
And the image the model sees is not raw. The wrist camera arrives with a two-centimetre world grid projected onto it, a crosshair at the pinch point, brackets showing the jaw opening, a line along the object’s axis, and a JSON of the robot’s state alongside. Google is rendering the “picture of the world” into the prompt by hand so the model doesn’t have to infer geometry from pixels alone.
The remaining demos are the frontier as an offline labeller (a four-level annotation hierarchy over a two-minute origami video, with retries down-weighted for training) and as an evaluator (dense progress curves over a sixty-second manipulation episode). What their post doesn’t mention is that every demo runs for seconds to a couple of minutes. None of them runs for an hour or more, and there isn’t a cost figure anywhere in sight.
And then the self-improvement demo gives the whole game away. One photo of a tabletop is used to create a simulation twin. A twin that generates 4,500 synthetic episodes. Those then train a small policy model, which lifts real-world success from 56% to 75%. The frontier model built the environment, compiled a cheaper model from it, and stepped out of the loop. Build, teach, judge - and then leave. That’s the heart of the entire post.
Astra: NVIDIA prints the bill
NVIDIA’s Seattle Robotics Lab directly asked the question.
Can (and should) a general-purpose model serve as a robot policy?
They tested GPT-6 Astra in three roles on 120 pick-and-place tasks.
Direct: the model emits actions from each observation, no memory.
Agentic: a persistent session that keeps its full history and has tools.
Orchestrated: a small robot model proposes actions and the frontier model reviews and intervenes.
An episode is one pick-and-place, roughly 25 seconds of robot time. The best result costs thirty million tokens and thirty-six dollars for that. The report’s own conclusion is that latency alone rules these models out as practical real-time controllers.
Where do all the tokens go? Into memory. The agentic policy’s edge comes from keeping its entire history (every plan, action and image) and re-reading it on every turn. Full history hits 100% on a subset - visual history alone manages 56.5%. The authors ran that ablation on a reduced task set because, without caching, it was too expensive to run in full. But the accuracy and the cost are the same thing. Context is the most expensive place that state can live.
Two more results also point in the same direction. Adding gripper and camera geometry to the prompt takes the Direct policy from 19.3% to 72.7%. The agentic policy barely moves with the same information, because it can reconstruct geometry from its history - at the cost of 14.7% more control steps. Hand the model the state and it stops paying to infer it. And three recent observations are enough - longer or denser history adds nothing. A short window plus explicit state beats a long window.
The Orchestrated policy is the picture from the start of this post, built and measured. A small model runs at 0.7 seconds, while the frontier reviews at a cadence that’s an explicit tuning knob, and the pair combined nearly doubles the small model’s success at half the agentic cost. The gap to the agentic policy is bounded memory - exactly what a maintained external state would close.
But the simulator pauses while the model thinks, so an 18-second call doesn’t prevent success - yet in the real world it likely would. And it’s important to recognise that NVIDIA sells the hardware that local inference runs on, so this framing clearly suits them. The data is published in enough detail to check for yourself. And one more detail worth noticing is that they report cost per successful episode. That’s cost per verified result, not cost per token, and it is provided without any commentary.
AA-3D: measuring the write side
Artificial Analysis’s new benchmark lives on the other side of the camera. Each model gets a written brief (create a: spinning black hole, a playable pinball table or the Colosseum) and it builds an interactive three.js web page in a sandbox with Blender, a physics engine, Python, a headless browser and a library of free assets. It iteratively writes code, renders, looks at its own screenshots and revises, for up to three hours or 250 turns. Then a human panel compares scenes blind and the models are ranked by Elo.
Claude Sonnet 5.5 and Opus 5.5 lead at around 1190, GPT-6 Astra is at 1165, and the field falls away fast below that. Two details matter more than the order. A mid-tier model ties the flagship, and in one vendor’s lineup the Flash variant beats the Max. So interestingly, the capability this benchmark measures seems to be driven by agentic discipline (look at your render, fix it, go again) more than by raw size.
In our bigger picture, this is the isolated “generation interface” - the frontier as builder. Argon’s sketch-to-simulation and its reproductions of robotics papers as browser physics labs are exactly this task, and now there’s an independent score for it. It’s also the economics in miniature - three hours of frontier compute once, and what comes out is a web page that runs forever at almost zero marginal cost.
But notice what the benchmark rewards. Panelists choose the scene they prefer. Nothing here checks that the mitochondrion has the right membrane structure or that the pinball obeys real physics beyond looking right. Generated worlds can be convincing before they are correct. For a game that’s fine. For anything a real system must trust and rely upon (a digital twin, a map, a training environment) the missing step is verification against reality, which is precisely the real2sim matching loop Argon demonstrated and AA-3D doesn’t score.
One small but very interesting signal in the metrics is that Artificial Analysis tracks how early each model looks at its own render, and that this correlates with quality. It seems that checking your work pays off, and now that is being measured.
Putting it together
If we line these three points up then the division of labour seems clear.
Build time is where the frontier model earns its keep - write the harness, build the twin, generate the edge cases, label the data, judge the hard calls. Expensive, occasional, and amortised across everything that follows.
Run time belongs to a maintained picture of the world, a loop of small models and deterministic code around it, and the frontier model only used as the escalation path.
This means that the snapshot of the world is the asset. And the models are tenants. Whoever maintains the state, the record of what happened and the means to check it owns the thing that doesn’t go out of date when next new model ships.
And the important metric that unifies all three is cost per verified result. NVIDIA called it out explicitly. While Google’s call budgets imply it. And AA-3D’s human votes are a very rough proxy for it.
The thing that none of the three measure is whether their picture of the world is right. AA-3D scores plausibility. Astra scores success in a paused simulator. Argon shows demos that worked. But correctness against reality is the gap in all three, and this is where the real work now sits.
But won’t the next model just fix this?
This is the obvious question. Frontier models get faster and cheaper every month and Sutton’s Bitter Lesson keeps marching ahead, surely the 18-second call becomes a two-second call and the loop collapses back into one model.
I believe it won’t, because speed and intelligence are two different things. A model’s latency at low batch is set by how many bytes need to stream through memory for every token, so the fastest thing you can run on a given piece of hardware is always the smallest good-enough model, not the smartest one. If you compress a 27-billion-parameter multimodal model to a few gigabytes and put it on frontier-grade hardware then you get answers faster, in tens of milliseconds. Put the frontier model on the same hardware and you get the Astra numbers. Diffusion-style decoders that emit whole structured outputs in parallel widen this gap even further for exactly the short, typed answers a perception loop needs.
So the good-enough model at real-time speed arrives before the frontier model gets cheap enough to sit in the loop, and that loop architecture is exactly what lets you use it in realtime. A better frontier model doesn’t dissolve this picture. It just becomes a better builder, teacher and judge above a loop that was already running without it.
Where this points
Here are my predictions, because that’s what they are.
The frontier model’s share of run-time calls in any deployed physical-world system will head downwards, while its share of build-time work goes up. Argon’s self-improvement loop is the template - build, compile, then step out.
The maintained picture of the world is the product layer in its own right (state, provenance and evaluation) independent of any model. And whoever owns it is model-agnostic by construction - and then any better frontier model is just a gift rather than a threat.
State leaves the context window. This one is less a forecast and more an accounting result. NVIDIA has already shown that keeping memory as re-read tokens costs thirty million of them per 25-second task, and that three observations plus explicit state does the same job as a long history.
Verification will become the benchmark. Expect the next generation of evaluations to score cost per verified result and correctness against ground truth, not preference.
The loop will move closer to the sensors. Latency, cost and privacy all point to small models on owned or nearby hardware doing the always-on work, with the frontier in the cloud handling exceptions and the residue.
So the frontier determines what’s possible, while the loop determines what’s affordable. And the state determines what’s true.
Build the understanding once, then make everything else a question.