What happened this week in AI by Louie
This week brought another strong set of model releases, with Meta moving ahead of OpenAI on several benchmarks. At maximum reasoning effort, Muse Wonder Full scores 56 on Artificial Analysis’s Intelligence Index, ahead of GPT-6 Astra at 53 and GPT-6.1 Sol at 52, although Opus 5.5, with fallback enabled, still leads at 58. Meta also reports strong results on longer coding projects. I would not read the 56 as Meta’s API model overtaking Astra, though. That score belongs to Wonder Full, which is only available through a Muse Code subscription; the standard Wonder model available through the API scores 52.
I am more interested in Meta’s emphasis on memory and persistence than a few leaderboard points. Its Active Memory system is designed to retrieve relevant context across conversations, and Active Focus is meant to keep an agent working through blockers and changes of direction. Those are key capabilities a personal agent needs when it has to work across several days. Meta is now a serious contender for demanding agent work, although I still want to see how well those memory and persistence claims survive our messy real projects.
We also got a long-overdue Haiku 5.5 update from Anthropic, an Ultrafast tier that OpenAI says generates GPT-6.1 Sol tokens up to eight times faster, and strong new Western models from Mistral and Reflection with open weights on the way. Mistral Large 4 is in API preview, and Beam is in selected early access, with both companies promising weights later this month. OpenAI’s Decisions API and Microsoft-Decision-1 also look aimed at the demand Jev exposed for fast, inexpensive model judgments in routing, classification, and scoring. There is plenty more on these releases in the news section below.
This week’s results are also a good reminder not to overread small benchmark gaps. Wonder Full at 56, Astra at 53, and Sol 6.1 at 52 look very close. In real work, the difference can be much larger. Larger frontier models are often better at handling unusual documents, conflicting instructions, incomplete tool output, and changing requirements without needing as much intervention. Averaging those cases into one score makes it easy to miss the failures that consume most of your time.
Part of this problem is benchmaxxing: concentrating training on the capabilities that raise benchmark scores while leaving a much more jagged level of intelligence elsewhere. At the bluntest end, that means training on the exact questions used in an evaluation. But I suspect the larger effect comes from building similar datasets and reinforcement-learning environments, repeatedly rewarding the model for the kinds of problems the leaderboard measures.
But benchmaxxing also improves real capabilities. Once we find a useful way to measure something, we give researchers a clearer target and a way to tell whether training helped. The problem is allowing that target to become our whole definition of intelligence. We need to keep expanding and refreshing our benchmarks as models improve. Running the same tests we ran two months ago helps track progress, but it’s insufficient for finding where the frontier has moved. I want both continuity and new tests that expose weaknesses the last training run had little reason to fix.
In mathematics and science, the next test may be the research itself. Instead of asking whether a model can solve a fixed problem, we can ask whether it can produce a useful result that was not already known. OpenAI’s recent release of hundreds of mathematical manuscripts shows both how promising and how difficult that is to evaluate. The collection includes proposed results at different stages of verification, with revisions and withdrawals already recorded. A proof that checks, a genuinely new theorem, and an important mathematical advance are three different achievements. Even so, testing models against unresolved research questions is a much more ambitious exercise than measuring another gain on a fixed exam.
Progress is less even in fields where quality is harder to verify. Writing is one of them. A paragraph can be grammatically clean, factually correct, and still contain nothing worth reading. A model can remove obvious slop words while preserving the generic argument, repetitive structure, and lack of a real point of view. Testing whether the text sounds fluent misses much of what makes it useful.
That is why we built ToneBench, led by my co-founder Louis-Francois Bouchard. We built it around a task we know extremely well: writing complete video scripts in Louis’ voice from a brief, research, examples, and detailed instructions. The current board covers 78 models across 127 configurations, with reasoning levels counted separately. Each configuration writes without tools, producing five drafts for each of ten real script tasks, giving us 6,350 drafts and 25,400 judge assessments. The tests cover sustained writing across a whole piece, where a strong opening can give way to repetition or a lost argument several paragraphs later.
The results are already useful for deciding which model and reasoning level to use. Opus 5.5 leads overall, but xhigh and maximum effort have the same rounded Writing Elo and overlapping uncertainty ranges. At API list prices, the average script costs about $1.00 at xhigh versus $2.58 at maximum effort, so I would use xhigh by default for this kind of writing. GLM-5.3 Flash at max ranks twelfth among the 78 models, each ranked at its best setting, at about 2.2 cents per script. At that price, I would test it for higher-volume work, but the generation cost is only useful if the drafts do not require substantially more editing. Measuring that human editing time is one of the next additions we want to make. For a consistent comparison, these costs include generation and reasoning at API list prices, even where the benchmark itself ran through subscriptions.
ToneBench is still a narrow benchmark. It tests one writer’s voice across ten tasks, so it cannot tell us which model is universally best at writing. Each draft is scored by four automated judges from Anthropic, OpenAI, Z.ai, and TypeSafe, with the model identity hidden. Those judges still bring their own preferences even when the model names are hidden. In our analysis, both the OpenAI and Anthropic judges rate outputs from their own labs more favorably relative to the other judges. Using several judge families helps reduce the influence of any one model’s preferences, but it does not remove the bias. Next, we’ll add more human ratings, and we are still preparing the public page for that.
Between ToneBench and the evaluation pipelines our deployment team builds for clients, we have been developing broader expertise in benchmark design. We plan to scale this in three directions:
Broader technical writing: expand ToneBench with new tasks and human feedback from contributors drawn from the roughly 5,000 writers who have published technical AI articles with Towards AI.
Business writing: evaluate client emails, executive summaries, proposals, and investment memos, where accuracy, selection, and judgment need to survive the move into a different professional voice.
Workflows and agents: build tasks modeled on work we deliver for private-equity-backed companies, including complex multistage video analysis, tender screening, responses to requests for proposals, and technical engineering image analysis. We want to measure output quality alongside human review time and cost, so the results can support credible return-on-investment estimates.
Please reach out if you would like to partner with us on continuing to develop Tonebench or building these future benchmarks.
Why should you care?
I would like AI labs to work much more closely with the people who know why an output that looks good still fails in practice. An experienced editor can see when a rewrite removes the detail that made the piece worth reading. An engineer can spot the unusual drawing that breaks an otherwise convincing image-analysis system. An investment professional can tell when a summary is accurate but too generic to help with an investment decision. That expertise becomes useful for training once we turn it into examples, criteria, and repeatable tests.
Defining those criteria requires just as much care. Reward polish too heavily, and the model may remove awkward but important details. Reward brevity without enough context, and it may drop conditions the reader needs. In writing, an unpopular opinion, an exception, or an unresolved question may be the most useful part of the piece. An evaluation must leave room for those choices rather than rewarding whatever automated judges find consistently clean and agreeable.
This gives practitioners a concrete way to influence what improves next. Keep examples of work that needed substantial correction and record why the correction was necessary. Include the time an expert spent getting it right. Those records expose capabilities that a general leaderboard misses, and give us something better to train and test against, and ultimately build solutions to in our systems.
— Louie Peters — Towards AI Co-founder and CEO
Today at AI Engineer in New York, we’re presenting ToneBench, our new benchmark for AI writing.
We tested 78 models across 127 configurations on ten real long-form script tasks. Each model received the same brief, research, voice guide, and constraints, then wrote every script five times. Four blind judges scored each draft across nine dimensions, giving us 6,350 scripts and 25,400 assessments.
The full leaderboard, methodology, judge analysis, cost comparisons, and model-vs-model pages are now public.
Hottest News
1. OpenAI Adds an 8x-Faster Ultrafast Tier for GPT-6.1 Sol
OpenAI added an Ultrafast API tier for GPT-6.1 Sol (and GPT-6 Astra), claiming speeds up to 8x faster than Sol Standard for latency-sensitive workloads. It uses the same model and capabilities but allocates faster inference through service_tier: “ultrafast”; OpenAI recommends WebSockets for agents making frequent tool calls so network overhead does not erase the latency gains. Ultrafast is available to all API users with separate rate limits and supports global processing plus US and EU data residency (Only US and global for Astra). In Ultrafast, Sol costs $12/M input and $60/M output tokens (six times the standard price), and Astra costs $60/M and $300/M for prompts up to 272K tokens.
2. Anthropic Releases Claude Haiku 5.5
Anthropic released Claude Haiku 5.5 for high-volume and cost-sensitive tasks. In the release notes, Anthropic also said it pairs well with Opus 5.5 and Sonnet 5.5 as a subagent on coding work. It is the first Haiku model with adjustable reasoning effort, and it substantially improves over Haiku 4.5, scoring 72.4% vs. 15.7% on the OSWorld 2.1 (offline subset) and 45.9% vs. 10.2% on Humanity’s Last Exam (without tools) in Anthropic’s evaluations. For prompts up to 100K tokens, pricing drops to $0.10/M input and $0.50/M output, 90% below Haiku 4.5; prompts above 100K cost $0.50/$2.50. Haiku 5.5 is available through Anthropic, AWS, Google Cloud, and Microsoft Azure, although Anthropic still recommends Sonnet or Opus for more complex agentic coding. Anthropic is also halving Claude Sonnet 5.5 cache-read pricing, which it says reduces the cost of most agentic workloads by around 20%, and introducing monthly Claude API credits for Max and Team subscribers.
3. Mistral Launches a Preview of Mistral Large 4
Mistral launched Mistral Large 4, a natively multimodal Mixture-of-Experts model with 1.05T total parameters and 52B active parameters. Mistral trained ML4 from scratch on 3,800 NVIDIA Grace Blackwell GPUs in its own European data centers, using multilingual data spanning more than 160 languages. The company reports 61.7% on DeepSWE v1.1, 28.3% on Terminal-Bench 4, and 42% on Dense 200. The model is available in public API preview through Mistral Studio at promotional pricing of $0.68/M input, $0.07/M cached input, and $2.09/M output tokens. Mistral plans to release the weights by the end of October after additional red-teaming.
4. OpenAI Publishes 722 Math Manuscripts From an Unreleased Model
OpenAI published 722 mathematical manuscripts across 372 result families produced largely by an unreleased internal frontier model while testing it on open research problems. The model attempted roughly 4,000 problems, with each retained result using the equivalent of about three hours of ChatGPT Pro thinking compute on average. The repository includes papers, source files, 10 abridged reasoning summaries, and Lean formalizations for many of the proofs. OpenAI cautions that the results are at different stages of verification and that some manuscripts without formal proofs may contain errors, with corrections and revisions to be tracked publicly.
5. OpenAI Releases the Decisions API in Public Beta
OpenAI released the Decisions API, a new endpoint that returns structured predictions instead of generated text. It accepts text and images and can return boolean predicates, fixed-set choices, or rubric scores alongside probabilities. OpenAI says responses are about 10× faster than its Responses API, although the public beta currently supports only GPT-6 Luna. Pricing is $0.10/M input tokens, with no output, cache-read, or cache-write charges. OpenAI says general availability is planned in the coming weeks.
6. Microsoft AI Releases Microsoft-Decision-1
Microsoft released Microsoft-Decision-1, a model built for structured tasks such as routing, classification, prioritization, verification, and workflow control. Microsoft reports that it achieved the highest accuracy in its 36-benchmark evaluation spanning nearly 150,000 questions, while its P50 latency was roughly 35× faster than GPT-6 Sol and 2.5× faster than H2O-Lightning-4B v1.1. The API also returns probabilities intended for threshold-based automation and human-review decisions. Microsoft-Decision-1 is available through Microsoft Foundry and OpenRouter at $0.042/M input tokens with no output-token charge.
7. Google Cloud Launches Gemini Agent
Google Cloud introduced Gemini as a single enterprise agent that can answer questions, create content, write and run code, and complete longer tasks across workplace systems. It runs persistently in the cloud, can create temporary subagents, use reusable skills and tools, maintain several forms of memory, and connect to systems, including Workspace, Microsoft 365, Slack, Salesforce, ServiceNow, Git, Jira, and MCP servers. The agent is separate from the underlying model: Google says it can dynamically route work across Gemini models and Anthropic’s Claude models, with more model providers planned. Google is also bringing the same agent into Gmail, Drive, Docs, Sheets, Slides, Chat, and Calendar and supports coworker agents with their own identity, email, storage, and permissions.
8. Reflection AI Introduces Beam
Reflection AI introduced Beam, a sparse Mixture-of-Experts model with 501B total parameters and 23B active parameters aimed at coding, reasoning, and agentic workloads. Reflection pretrained it on 23.8T tokens and then ran more than 100 million RL rollouts on 10,500 NVIDIA GB300 GPUs over four weeks. The company reports scores of 44.4% on DeepSWE v1.1, 77.2% on SWE-Bench Pro v2-Hard, and 80.1% on Terminal-Bench 2.1. Beam is currently undergoing final red-teaming and is available only to selected early-access users. Reflection plans to release the weights, technical report, model card, evaluation tools, and fine-tuning stack later this month under Apache 2.0.
AI Tip of the Day
Most human-in-the-loop systems ask someone to review the final output. That is often the worst point to catch a mistake. A long research report takes time to verify properly, and by then the agent may already have searched, read, summarized, and written from the wrong starting point.
In any agent workflow, place human approval before the steps that are costly to undo.
For a research agent, that checkpoint is often the search plan. A weak query can still return relevant-looking sources, produce a coherent summary, and lead to a conclusion that doesn’t answer the original question.
In Nova, the deep research agent we build in Agent Engineering, each round proposes five search queries, each with a one-line reason. The agent can pause there so you can approve the set, remove queries, or replace one before any search runs.
The reasons speed up review because you can check what the agent intends to learn from each query, not just whether the wording looks plausible.
A useful rule for future projects: put checkpoints where an error is easy to catch and expensive to carry forward.
Five 5-minute reads/videos to keep you learning
1. RAG vs. Long Context in 2026: When Retrieval Still Wins
The article examines whether million-token context windows have reduced the need for RAG. It uses RULER, Chroma’s Context Rot research, and Anthropic’s contextual retrieval work to explain how recall, cost, and latency change as prompts grow. It then covers cases where full context works better and provides a decision framework based on knowledge-base size, update frequency, call volume, and whether a task needs targeted retrieval or whole-document reasoning.
2. A Week Inside Jev: What a Non-Generative Decision Model Is Actually Good For
The author explores what TypeSafe’s Jev can actually do compared with the “200× faster, 400× cheaper” framing around it. He explains Jev’s Choice, Score, and Noul question types, then builds a dry-run Claude Code context pruner to show how a probabilistic decision model can fit into an existing workflow. The article also examines Jev’s limitations, available calibration evidence, and potential uses such as PR triage, support routing, and on-call paging.
3. Vector Search Is Not Enough: The Blueprint of GraphRAG
The article explains how GraphRAG uses knowledge graphs to retrieve relationships that ordinary chunk-based vector search can miss. It first covers how knowledge graphs are structured and built, then walks through retrieval approaches, including Cypher templates, dynamic Cypher generation, Text2Cypher, Graph-CoT, lexical graphs with vector search, and Microsoft’s community-based GraphRAG and DRIFT search. It closes with the cost, latency, and implementation trade-offs that affect which approach fits a given use case.
4. Jev and RLCD: Architecture, Open-Source Models, and Practical Agent Use Cases
The article explains Jev’s typed decision interface and how RLCD trains models to produce calibrated probability distributions. It separates Jev’s documented API from its undisclosed internal architecture, then examines two independent implementations, eve-rlcd and OpenJev-RLCD. From there, it shows how decision models can handle routing, RAG evidence checks, tool selection, and other bounded choices inside agent workflows, with local examples and guidance for evaluating calibration and end-to-end performance.
5. Muon: The Orthogonalized Optimizer Explained Through Equations, Intuition, Code and Infographic
The article explains Muon step by step, starting with how optimizers update weight matrices and what Adam’s per-coordinate scaling misses. It then derives Muon’s update rule and explains why orthogonalization gives the steepest-descent step under the spectral norm. From there, it covers how Newton–Schulz makes the operation practical without an SVD, where Muon fits into a Transformer, and how to implement it alongside AdamW. It closes with limitations and considerations around pre-training, distributed training, and using Muon outside the settings where it has been best tested.
Repositories & Tools
1. nanoMuse is a personal agent that runs across your phone, computer, and browser, completing tasks and asking before anything irreversible.
2. REA is an MCP server and CLI that lets coding agents reverse engineer binaries, apps, and websites locally.
3. RRSI is a framework for evolving agent harnesses while regularizing the search to avoid overfitting.
4. LongLive is NVIDIA’s NVFP4 parallel infrastructure for training and running long video generation models.
5. STEPQuant is a quantization framework that compresses the recurrent states of Delta-rule linear attention models, closely matching full-precision accuracy at 6 bits.
Top Papers of The Week
1. Memento 3: Rulebook Harness Reports a Perfect ARC-AGI-3 Score
Agents in unfamiliar environments need to figure out how the world works, but a few observations can fit several different explanations. Memento 3 lets a frozen LLM agent keep a written rulebook of its current guesses about how the environment works, then turn that rulebook into code it uses to predict and plan. As it plays, the agent compares predictions with what actually happens and revises both the rules and the code. The agent updates its code only if the LLM judges it faithful to the rulebook and it exactly reproduces past observations. On ARC-AGI-3, the agent clears every level of all 25 public games with a mean RHAE score of 100.0 while using 44% of the human action count.
2. PluginRSI: Recursive Harness Improvement With Reusable Plugins
Current harness optimization methods search over whole programs, which makes it hard to isolate and reuse the parts that actually help. PluginRSI breaks a harness into small plugins, improves each one separately, stores them in a shared library, and recombines them into new harnesses each round. It outperforms existing harness optimization methods on software engineering, command-line, and question-answering tasks. The resulting harnesses keep their advantage when moved to other models without retuning, and the plugin library speeds up optimization on unseen tasks.
On-policy distillation trains a student to imitate a teacher on the student’s own outputs, and some variants try to push the student past the teacher. Doing this through the model’s output probabilities is noisy and destabilizes training. The authors observe that RL training shifts a model’s internal representations in a measurable direction at every layer. Their method, RIDE, measures how the RL-trained teacher differs from its pre-RL version and pushes the student’s internal states slightly beyond the teacher in that same direction. Across four base and RL-teacher pairs, RIDE approaches or exceeds the RL-trained teacher on every pair, and consistently beats output-space extrapolation.
4. AgentGarten: Code Worlds for Evolving Agents
Agents learn from the environments they practice in, but building environments that are both rule-consistent and visually realistic remains hard. AgentGarten pairs simulators and game engines, which handle world state and rules, with a shared neural renderer that produces realistic visuals in real time. The renderer is adapted from a pretrained video model and trained with a new method called Adversarial Forcing. Agents explore these worlds and write what they learn into playbooks that later agents inherit and refine. The authors report agents learning in just 4 rounds, compared with millions for a conventional reinforcement learning approach.
Quick Links
1. OrcaRouter releases OrcaCyber Zero 1.5, the successor to OrcaCyber Zero 1.0, post-trained for authorized vulnerability reproduction, exploit development, and penetration testing. It ships with a 1M-token context window, 128K max output, native function calling, and structured outputs. OrcaRouter reports 100% on Cybench (39 of 40 tasks), 95.8% on a 24-task CVE-Bench subset, 93.9% on HumanEval+, and 76.5% on SWE-bench Pro V2. Access is gated to OrcaRouter’s Security Research tier via an OpenAI-compatible API at $3.00/$7.50 per million input/output tokens, with no weights released.
Who’s Hiring in AI
Forward Deployed Engineer Intern @Coinbase (New York, NY, USA)
Cloud & AI Engineer @IBM (Bangalore, India)
Artificial Intelligence Engineer @L3Harris (Melbourne, FL, USA)
AI Prompt Engineer @CACI International (Ashburn, VA, USA)
AI Engineer 5 @Capital One (Multiple US Locations)
Software Engineer @NetApp (Bellevue, WA, USA)
Interested in sharing a job opportunity here? Contact sponsors@towardsai.net.
Think a friend would enjoy this too? Share the newsletter and let them join the conversation.
Who knew guessing five-letter words could become such an enjoyable habit? Wordle Unlimited has a simple design, clear feedback, and just enough difficulty to keep me interested. I especially like trying different starting words to see which ones work best. https://wordleunlimitedtoday.io
It's fascinating how these benchmarks can impact everything from ethical standards to tech innovation. Thanks for shedding light on such a crucial issue!
https://cargamesfree.io