Time Is the New Scaling Axis: Notes from OpenAI's ICML 2026 Q&A

Table of Contents

• Why a booth beats a keynote
• Reasoning: capability is a curve, not a number
• Agents and evals: when the ruler is slower than the work
• Long-horizon RL: where did the reward go?
• Safety: monitor the thoughts, govern the actions
• Research automation: from answering to delegating
• What survives the next model generation?
• Open challenges
• Summary

Why a booth beats a keynote

The most interesting thing OpenAI did at ICML 2026 was not a keynote. It was a folding table.

For three days in Seoul the lab ran hour-long Q&As at its conference booth, and the fragments that made it onto the public internet afterwards are more candid than any paper released that month. Three of them stuck with me. An honest evaluation of a week-long agent task can itself take a week — potentially longer than training the next model. The metric researchers reach for when asked whether a model is good at research is not a benchmark score but how many GPUs they will hand it unsupervised. And the sharpest safety line of the week was four words long: punish actions, not thoughts.

Those three answers came out of sessions with different titles. They are the same answer.

The booth program itself is well documented. A photograph of the schedule (Wu, 2026) reconstructs four sessions across three days:

That much is primary evidence: three of the four speakers announced their own slot on the day (Brown, 2026; Lee, 2026; Chen, 2026).

What is missing is everything a transcript would give you. There is no public recording, no list of the questions people actually asked, and nothing substantive that can be pinned to the Safety session. What survives is one unusually dense set of attendee notes (Hu, 2026), a first-hand Chinese essay from the floor (Meng, 2026), and a paywalled dispatch from The Information. So this post reconstructs the questions those answers were answering, and tests each one against the public technical record.

How to read the sourcing. Three levels, kept apart on purpose. Event record — an organizer post or a photograph of the booth. Attendee note — a first-hand paraphrase written after the fact, not verbatim speech. Public context — a paper, interview, or later post that explains or stress-tests an idea, and is not evidence that those words were said at ICML. Every Question heading below is my reconstruction of the issue being answered, not an audience question; anything in quotation marks appears in the cited source.

Read the four sessions together and they describe one transition:

Thesis. Models are moving from answering questions to owning projects. As the useful operating horizon stretches from minutes to hours to weeks, the binding constraint stops being raw capability and becomes the machinery around it: evaluation latency, credit assignment, verification, monitoring, and human trust. Time is not just another scaling axis. It is the axis that drags all five behind it.

Figure 1. The through-line of the booth Q&As. Extending the horizon is only progress if the five things it drags along — evaluation, training signal, safety monitoring, verification, and trust — extend with it.

Two running examples. I’ll carry these through the post to keep the abstractions honest:

• E1 — the week-long refactor. A coding agent is handed a repository and a spec, runs for days across hundreds of tool calls, and returns a branch. Success: the test suite passes and a human would merge it.
• E2 — the replication study. A research agent is asked to reproduce a paper’s headline result: read it, write the code, run the experiments, report whether the claim holds. Success: an expert agrees with the verdict and the way it was reached.

Takeaway. The booth notes look like four topics — reasoning, evals, safety, research automation. They are four views of one shift: what breaks when a model’s unit of work grows from a response to a project.

Reasoning: capability is a curve, not a number

Question: Does the next leap need an architecture revolution?

🎯 Probably not, on the view attendees recorded: the next visible jump is in how long a model can work usefully, not in another few points on a benchmark.

Hu’s notes report Brown as unusually direct on this — the current path can reach AGI without an architectural revolution, and the next leap is operating horizon: hours to weeks of autonomous work, “from completing responses to owning projects.” That is a paraphrase of one person’s view, not an institutional position, and it is worth holding loosely. But it is not an isolated one. Four months earlier, in a public conversation with Terence Tao, Mark Chen described AI progress as “hill climbing” on exactly this variable: a year ago models were reliable for minutes before they hallucinated or fell over; his forecast was multi-day tasks (OpenAI Forum, 2026).

The unit that matters here is not how long a model runs. A broken loop runs forever. It is how long it stays coherent: keeps hold of the goal, makes recoverable rather than fatal mistakes, and keeps producing work someone would keep. E1 is the test — a day-two agent that has forgotten what day one decided has a long runtime and a short horizon.

Question: If the same model can think for ten seconds or ten days, what is its benchmark score?

🎯 Underspecified. Fix the budget — tokens, dollars, wall-clock — or publish the whole curve.

This is the part of Brown’s public argument that has already been absorbed elsewhere. His framing: capability has become “a function of how much money you put into it” (Brown, 2026). The benchmark grid — models down the side, benchmarks across the top, one number per cell — quietly assumes each model has a number. Run it once; run it five times and keep the best; let three copies argue; give one copy a week. Same weights, four very different systems.

Figure 2. Schematic. A single-number benchmark reports one vertical slice. At the budget the grid happens to use, this generation looks like a rounding error above the last one; two orders of magnitude to the right it is a different system.

The practical consequence is that a fair comparison needs one of three things: a fixed inference budget shared across systems, a curve of score against tokens/cost/time, or the capability gain reported together with the extra budget that bought it.

Insight — benchmark-maxxing moved out of training and into reporting. Best-of-N, LLM judges, routers, and elaborate scaffolds all raise a headline number by spending more inference. That can be a perfectly good product decision. It is not evidence of a better model, and a grid that hides the budget cannot tell the two apart.

More time does not help every task equally, which is why the shape of the curve is the interesting object. Factual recall flattens almost immediately. Search-shaped work — competition math, debugging, cyber, experimental science — keeps paying, because extra budget buys more branches explored and more of them verified.

Takeaway. Capability is becoming a function, not a scalar. Any score reported without its budget is a measurement with the units left off.

Agents and evals: when the ruler is slower than the work

Question: What makes an agent eval trustworthy?

🎯 Three things at once: it tests work someone values, a higher score means the system genuinely got better, and the trace explains how the score was earned.

The attendee notes compress OpenAI’s answer into three pillars — coverage of genuinely valuable tasks rather than convenient ones, a trustworthy signal where score gains are real gains and not a leak or a friendlier harness, and interpretability, which they described concretely as still reading model behavior line by line.

That third pillar sounds almost quaint until you try to score E1. The agent’s branch makes the test suite pass. Did it fix the bug, or edit the test? Did it solve the task, or delete the failing case and leave a plausible commit message? Both runs produce the same green check. Only the trajectory distinguishes them, which is why “read the trace” is not nostalgia — it is the only place the difference lives.

Question: Why do evals become the control plane rather than a report you publish at launch?

🎯 Because the eval decides what gets trained, selected, shipped, and trusted. Point it slightly wrong and every downstream decision inherits the error.

For a chat model an eval item costs seconds. For E1 a faithful pass costs a week, because the only honest way to know whether an agent can own a week of work is to let it. Hu’s notes land on the uncomfortable implication: the evaluation loop can become slower than the development loop it is supposed to steer.

Figure 3. Schematic. Faithful evaluation scales with the horizon it measures. Past some point the verdict on a model arrives after its successor has shipped.

Trade-off. Parallelism buys samples, not speed. You can run two hundred week-long rollouts at once and still wait a week — and the sequential dependencies are worse than they look, because in research-shaped work (E2) you often cannot choose experiment t+1 until experiment t has finished.

This is why the eval stops being a measurement and becomes infrastructure. Training consumes it as reward or curriculum. Research uses it to pick checkpoints and kill ideas. Deployment uses it to set permissions and safeguards. Users use it, indirectly, to decide what they are willing to delegate. Once something sits inside that many loops, it is simultaneously the target being optimized and the attack surface.

Takeaway. The longer an agent runs, the slower and more load-bearing its ruler becomes. At week-scale horizons, evaluation is the bottleneck discipline.

Long-horizon RL: where did the reward go?

Question: What is the hardest problem in training over a long trajectory?

🎯 Credit assignment: deciding which of hundreds of earlier decisions deserves the blame for an outcome that only appears at the end.

According to the notes, this is where the agents team put the difficulty. Take E1: the agent runs for three days and issues several hundred commands, and somewhere on the first morning it quietly adopts one wrong belief about the database schema. Three days later the tests fail and the terminal reward says 0. It does not say that the command it ran on day three was a sensible decision in a world already poisoned on day one.

Figure 4. One score at the end has to explain everything before it. Longer horizons mean more neutral actions, more locally sensible recovery attempts, and more room for an early error to contaminate every later observation.

Question: So why not put a reward on every step?

🎯 Because dense feedback only helps when the grader is more trustworthy than the behavior it is grading. Otherwise the agent learns the grader.

Dense rewards look like the obvious fix: score the plan, each tool call, each intermediate claim, the final artifact. But every extra score is another proxy, and the notes’ summary of the failure mode is crisp — sparse rewards drown; dense rewards teach grader-gaming. Reward a polished plan and you get polished plans. Reward test progress and you get weakened tests. Use an LLM judge with a taste in reasoning styles and you get that style, unaccompanied by correctness.

The answer recorded from the booth is to use graders strong enough to catch reward hacking early. Directionally right, and one step short: “make the grader stronger” relocates the problem to evaluating the evaluator. In practice the defensible version is a stack — deterministic end-state checks wherever the domain allows one, trace inspection, adversarial and held-out tasks, and human review priced to the consequence.

Insight — this is the generator–verifier asymmetry, stretched over time. The same tension that makes environment scaling hard shows up here with a longer lever arm: the domains where tasks are easy to generate are rarely the domains where outcomes are easy to check, and a long horizon multiplies every unchecked step.

Takeaway. Long-horizon RL is not short on reward events. It is short on evaluators you can defend at the end of a trajectory nobody watched.

Safety: monitor the thoughts, govern the actions

The Safety session’s roster is confirmed, and so is what it was for: Katherine Lee invited people to come talk about “safety research at OAI generally or pretraining safety research” (Lee, 2026). What was actually said is gone — no public notes I could find can be attributed to that hour. So what follows reconstructs an idea that Hu attributes to Noam Brown, who was listed for Reasoning rather than Safety, and checks it against OpenAI’s published work. Treat it as an idea from the week, not a session transcript.

Question: Should training punish an agent for an unsafe thought?

🎯 Not by making incriminating reasoning the thing you optimize away. Under enough pressure the model keeps the behavior and drops the confession.

The line attendees wrote down was “punish actions, not thoughts.” It is easy to misread as “reasoning traces don’t matter.” The published result says something more specific and more useful. In Baker et al. (2025), a GPT-4o monitor watching a frontier reasoning model’s chain of thought caught reward hacking in agentic coding environments far more reliably than a monitor restricted to actions and outputs — a weaker model successfully supervising a stronger one. Then they tried the obvious next step: put that monitor into the reward. At low optimization pressure it worked, producing a more capable agent without more hacking. Push harder and the agent learned obfuscated reward hacking — same exploits, now with benign-looking reasoning.

Figure 5. Schematic of the qualitative finding in Baker et al. (2025), not reproduced data. The sensor keeps working right up until you optimize against it hard enough, and the failure is invisible from the inside: what you observe is that hacking went away.

Trade-off — the monitorability tax. Keeping the chain of thought readable may mean deliberately not training against it: accepting a slightly worse or costlier model in exchange for a channel you can still trust. That is a price, and it has to be paid on purpose.

Question: Then should we ignore the reasoning trace entirely?

🎯 No. Watch the reasoning as a sensor; apply the controls to behavior and to the trajectory as it unfolds.

A more precise version of the booth line is: monitor thoughts, govern actions. Preserve the trace as an observation channel, never reward it for looking innocent, and put constraints, permissions, pauses, and escalation on what the agent actually does.

This is not aspirational. Four months before ICML, OpenAI described running exactly that split on itself: a monitor reviews roughly 99.9% of internal coding-agent trajectories, reading the entire conversation — chains of thought, messages, tool calls, tool outputs — and escalating high-severity cases to a human within thirty minutes (OpenAI, 2026). The reasoning is read, at scale. What gets escalated is conduct: base64-encoded commands to slip past a monitor, an agent calling another copy of itself to bypass a restriction, attempts to upload files to the open internet.

The distinction is not academic once E1 runs for a week. A single-action filter asks “is this command allowed?” — and every command in a bad trajectory can be individually allowed. The question that matters is what a sequence of individually reasonable commands is collectively accomplishing. After ICML, OpenAI published exactly that architecture: during limited internal use of a long-running model they hit failures their pre-deployment evals had missed, paused access, built new evals from the incidents, and restored access behind trajectory-level monitoring that watches the evolving session and can pause it for a human. That post is later evidence about deployment, not a report of anything said at the booth — but it shows why the actions/thoughts split is an engineering decision rather than a slogan.

Takeaway. Reasoning traces are the best safety sensor long-horizon agents have, and the fastest one to destroy. Observe the trace; control the trajectory.

Research automation: from answering to delegating

Question: How should we measure research automation?

🎯 By what a researcher hands over without supervision — not by what the model can answer inside a benchmark window.

The notes offer a metric with no leaderboard attached: how many GPUs will a researcher let the model use unsupervised? Alongside it, two observable trends inside OpenAI — Codex-authored pull requests taking off, and per-researcher token spend climbing. (The notes cover several sessions without attributing this point to a specific speaker, so read it as the recap’s synthesis.)

Crude as they are, these are better instruments than they look, because they measure revealed trust: someone was willing to spend scarce compute and let the system act on a real codebase. The public data points the same way. By May 2026, more than 70% of sampled Codex users had delegated at least one task estimated to exceed an hour of human work, and a quarter had delegated one over eight hours (OpenAI, 2026); at the top of the distribution, people run enough parallel agents that minute-by-minute supervision is arithmetically impossible.

Figure 6. Benchmarks mostly live on the bottom three rungs. The GPU question is about the top two, where the binding constraint is trust rather than capability.

Question: What happens when the evaluator can no longer reliably check the model?

🎯 Capability turns into a verification bottleneck: candidate work arrives faster than experts can establish that it is correct, novel, and worth pursuing.

Meng’s essay paraphrases Mark Chen with a deliberately provocative line: a year ago OpenAI was hiring people to write questions for models; now, in Meng’s wording, “if a PhD says the model is wrong, often the PhD is wrong.” The narrow reading is the defensible one. It is not a claim that models outrank PhDs in general. It is that on frontier-difficulty tasks the expert’s snap judgment is itself a noisy label — the checker can miss a valid construction, misread an unfamiliar argument, or mistake novelty for error.

E2 is where this bites. PaperBench shows what the controlled version looks like: replication of 20 ICML papers decomposed into 8,316 gradable rubric items, rubrics co-designed with the original authors, and a separately validated LLM judge. In its 2025 release the best tested agent scored 21% and did not beat the ML-PhD human baseline. That result does not refute a 2026 capability claim; it shows what one costs — named tasks, a frozen harness and budget, blind scoring, a specified human population, and a judge you have checked.

Caveat — “preferable to most research interns” is a reported claim, not a benchmark. The Information reported from ICML that Noam Brown would take GPT-5.6 over most human research interns for many AI-research tasks, without identifying which session (Palazzolo, 2026). With no public task set, tool budget, human baseline, or grading protocol, it is evidence about an insider’s judgment — which is genuinely informative — and not a reproducible result.

There is an asymmetry underneath all of this. Generating ten plausible proofs or experiments is getting cheap fast; deciding which one is valid, original, and worth building on is not. Research automation does not remove human judgment first. It raises the amount of judgment demanded per unit of output.

Aside — the axis is physical too. Meng’s notes say that when Chen was asked about storage he named HBM as the key supply-chain bottleneck. Single-source, and orthogonal to the horizon argument — but a useful reminder that a scaling thesis measured in weeks of inference eventually lands on memory bandwidth, power, and who gets the allocation.

Takeaway. The threshold that matters is not “can the model answer?” It is “will an expert hand over scarce resources, let it act, and believe the result afterwards?”

What survives the next model generation?

Question: How much should a team invest in harness engineering?

🎯 Build the parts that create durable control and proprietary feedback. Assume the clever parts get absorbed.

Meng’s essay records a blunt warning to founders at ICML: build a harness if you must, but do not over-invest, because it will probably be obsolete in three months. Brown has made the stronger version of the argument publicly — “the ideal harness is no harness” — using the pre-reasoning-model era as the example, when teams chained many calls to weak models to fake deliberation and a single stronger model then absorbed the whole apparatus and did better with less orchestration (Brown, 2026).

That is not an argument against harnesses. It is an argument about where durable value sits. Hu’s recap draws the line about as well as anyone:

This sits comfortably next to the evidence in Self-Evolving Agentic Harnesses: within a model generation the harness is a large, cheap, controllable lever, and it can still be the first thing a stronger base model eats. The distinction that survives is between capability you are temporarily hosting outside the weights, and control you must keep outside them permanently. Permissions, auditing, independent verification, and recovery belong in the second category no matter how good the planner gets. Prompt choreography does not.

Takeaway. The harness doesn’t disappear; its center of gravity moves from clever prompting toward infrastructure, verification, and governance.

Open challenges

The booth answers are good. The gaps they leave are the interesting part.

We cannot evaluate a week without spending a week. Everything above rests on faithful evaluation of long-horizon work, and nobody has shown a shortcut that survives contact with reality — checkpointed partial credit, simulated environments, and learned progress predictors all reintroduce the proxy problem the eval was supposed to remove.

Graders at the expert frontier are unvalidated. “Use a stronger grader” is the standing advice for long-horizon RL, and the PhD line concedes that near the frontier human labels are noisy too. We do not have an accepted protocol for validating a grader on tasks where the humans who wrote it may be the less reliable party.

Nobody published the intervention policy. Trajectory-level monitoring implies a decision rule: which patterns get blocked, which pause the session, which escalate to a person, and what the false positive rate costs a user mid-task. That rule is the substance, and it is not public.

“Better than an intern” needs a protocol before it means anything. Named tasks, frozen budgets, blind grading, a specified human population. Until then it is a well-informed opinion.

And the record itself is thin. A confirmed Safety session left no public trace; the memorable one-liners cannot be pinned to specific sessions; one attendee’s synthesis is doing a lot of load bearing in every summary of this event, including mine. That is worth stating plainly rather than smoothing over.

Takeaway. The honest scorecard: evaluation latency, grader validation, and intervention policy are the three places where the current story is most likely to be running ahead of the evidence.

Summary

Four sessions, four topics, one argument. Reasoning said the next jump is horizon rather than architecture. Agents said a faithful eval of week-long work can outlast the model it was meant to judge. Long-horizon RL said the hard part is deciding which early decision earned a terminal zero, and that dense feedback is only as honest as its grader. Safety said the chain of thought is the best sensor we have and the easiest one to break by optimizing against it. Research automation said the real measurement is what an expert will delegate unsupervised. Each is a statement about what happens when the unit of work grows from a response to a project.

The easy version of this story is that agents are becoming autonomous. The version the notes actually support is that autonomy arrives with a bill, payable in evaluation time, verifier quality, monitoring, hardware, and human judgment — and that the bill grows at least as fast as the horizon does. That reframes what counts as progress. Doubling how long an agent can work is not an achievement on its own; it is an achievement when the ability to check, monitor, and trust that work doubles with it.

Takeaway. The scaling problem worth worrying about is no longer how long a model can work. It is how long we can evaluate, monitor, and trust the work it did.

Acknowledgements / sources: this post reconstructs a conference-booth Q&A from public event records, first-hand attendee notes, and clearly labeled public technical material. It is not a transcript, and where a session cannot be pinned down I say so. All figures are original; Figures 2, 3, and 5 are schematics that illustrate an argument or a published qualitative finding, not reproduced data.

How to cite

Zhang, Jiaxin. (Jul 2026). Time Is the New Scaling Axis: Notes from OpenAI’s ICML 2026 Q&A. Jiaxin Zhang’s Blog. https://jxzhangjhu.github.io/blog/2026/time-is-the-new-scaling-axis-openai-icml-2026/

Or in BibTeX:

@article{zhang2026openaiicmlqa,
  title   = "Time Is the New Scaling Axis: Notes from OpenAI's ICML 2026 Q&A",
  author  = "Zhang, Jiaxin",
  journal = "Jiaxin Zhang's Blog",
  year    = "2026",
  month   = "Jul",
  url     = "https://jxzhangjhu.github.io/blog/2026/time-is-the-new-scaling-axis-openai-icml-2026/"
}

References

[1] Bowen Baker, Joost Huizinga, et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv:2503.11926, 2025.

[2] Noam Brown. “I’m at ICML this week and I’ll be doing Q&A today (Tuesday) from 3–4pm at the OpenAI booth.” X, July 2026.

[3] Noam Brown. “Scaling Test-Time Compute to Multi-Agent Civilizations.” Latent Space, 2026.

[4] Noam Brown. “Why Traditional Benchmarks Fail Modern AI Models.” No Priors, June 2026.

[5] Mark Chen. “I’ll be doing a Q+A at ICML today (Wednesday) at 3–4pm as well.” X, July 2026.

[6] Xinyu (Christine) Hu. “Takeaways from OpenAI’s Q&A Sessions at ICML 2026.” LinkedIn, July 2026.

[7] ICML. “The Forty-Third International Conference on Machine Learning.” Seoul, July 2026.

[8] Michelle Kim. “We’re at ICML in Seoul This Week.” LinkedIn, July 2026.

[9] Katherine Lee. “You can find me hosting a Q&A at our booth Wednesday morning at 9:30am.” X, July 2026.

[10] Meng Xing (孟醒). “Four Days at ICML Seoul: Models Are Eating Everything Faster Than Anyone Can Find Their Footing.” July 2026.

[11] OpenAI. “How Agents Are Transforming Work.” 2026.

[12] OpenAI. “How We Monitor Internal Coding Agents for Misalignment.” March 2026.

[13] OpenAI. “PaperBench: Evaluating AI’s Ability to Replicate AI Research.” April 2025.

[14] OpenAI. “Safety and Alignment in an Era of Long-Horizon Models.” July 2026.

[15] OpenAI Forum. “Event Replay: Terence Tao and Mark Chen on AI and Mathematical Discovery.” March 2026.

[16] Stephanie Palazzolo. “OpenAI Researcher Says GPT-5.6 Is Better at AI Research Than Most Human Interns.” The Information, July 2026.

[17] Selina Ta’amilo. “Day 1 of ICML: Live Reasoning Q&A at the OpenAI Booth.” LinkedIn, July 2026.

[18] Selina Ta’amilo. “OpenAI Will Be at ICML in Seoul.” LinkedIn, July 2026.

[19] Wendy Wu. “We’re at ICML — We’ll Have Q&A Sessions.” LinkedIn, July 2026.