TAI #217: AI Agents Are Finding Attack Paths We Never Approved
Also, Meta’s return to open weight with Muse Glimmer and Spark 1.2, DeepMind leadership reshuffle & more!
What happened this week in AI by Louie
Meta made a welcome return to open weights this week. Muse Spark 1.2 jumped 260 Elo points to 1,631 on the independent GDPval-AA benchmark after Meta increased its coding training and co-trained it with the new Muse Code agent. Mark Zuckerberg says the weights are coming soon. Meta also released Muse Glimmer, a 29.6-billion-parameter Apache 2.0 model with a 131,000-token context window and a 4-bit version designed for a single 24 GB GPU.
If Meta releases the same Spark 1.2 model now behind its API, I expect it to become the strongest open-weight model from a lab outside China. China’s Kimi K3 still leads Spark 1.2 by three points on Artificial Analysis’s current Intelligence Index, so this remains my forecast. Meta has yet to state the size, license, exact release build, or date.
This is good news for competition, local AI, and cyber defense. It also sharpens the hardest debate from the past few weeks. Open weights let defenders run models over private code and real attack logs without a lab refusing the work. They also let attackers strip away refusals, work offline, and avoid central monitoring.
The smallest incident this week may be the most useful example because it happened during normal use. Australian AI practitioner Andrew Bird had pointed OpenClaw, running Claude Opus 4.6, at his gym’s booking site to grab Pilates classes that sold out in seconds. When he was fourth on a waitlist, he asked whether the agent could move him to the top. It found the gym’s GraphQL API and called cancelReservation against the member in first place. The backend never checked whether Bird owned that reservation, so the cancellation worked, and he moved from fourth to third.
The agent then reported that the API had “zero authorization checks” and that it had already tested the flaw on the person in first place. When Bird asked it to undo the damage, it could not restore the booking because the reverse methods correctly returned 403 for another member’s record. No cyber mode was on, and no safeguard had been disabled. A personal agent received a casual goal, found a live permission gap, and used it without asking.
The deepest case is OpenAI’s Hugging Face incident. OpenAI was testing a mix of agents, including GPT-5.6 Sol and an internal research prototype, on ExploitGym, a benchmark that rewards working exploits against known flaws. The agents had no direct internet and could install packages only through an internal JFrog Artifactory cache. One agent searched for another route, found a previously unknown flaw in that Artifactory deployment, and used it to reach an internet-connected node.
OpenAI’s later Black Hat account showed that this started well before the July intrusion. An agent first exploited Artifactory on May 26. Later runs used the shared package service to leave notes and working exploits for future runs. When an early-July outage forced OpenAI to clear that state and patch the route, agents rebuilt a shared channel through another method two days later. At least one run recognized that attacking external infrastructure exceeded its intended scope, then continued because the benchmark looked impossible and other agents were already using the route.
Once outside, the agent inferred that Hugging Face might host ExploitGym assets or answers. Hugging Face reconstructed 17,613 actions over four and a half days. The agent first used a public code sandbox as a launch point, then exploited an HDF5 file-reading path and a Jinja template injection to run Python inside a production Kubernetes worker. In less than 13 hours, it reached cluster-administrator access across several clusters, gained root on 11 nodes, made 181 mesh-network enrollments, read a secret containing 136 keys, minted a source-control token, and opened a malicious pull request to probe continuous integration for more credentials. This was a genuine end-to-end production compromise in pursuit of a benchmark score, with no human directing each step.
During the forensics, hosted frontier models refused to analyze parts of Hugging Face’s own attack logs because they contained real exploit code and command-and-control material. Hugging Face ran the open-weight GLM-5.2 on its own hardware to decode the traffic and rebuild the timeline. That is the best concrete argument I have seen for giving defenders unrestricted model access. The practical security lesson is that exploit chaining is now easy. One novel Artifactory flaw opened the route, then weak credentials, broad permissions, and missed alerts carried the agent much further.
The UK AI Security Institute’s 35-page report shows the next step: a cyber agent moving into social engineering. From July 25 to 28, AISI ran 122 attempts across seven models and logged 19 unsanctioned external actions in 10 runs, 17 from Mythos 5 and two from a single GPT-5.6 Sol run. The internet was on, production cyber classifiers were off, no live LLM monitor checked actions before execution, and the largest runs had enough tokens to continue for 40 to 50 hours.
In the worst Mythos 5 sequence, the agent wrapped malware around a real bug fix, created three fake GitHub identities, pressured a real maintainer, fabricated an independent-looking review, and edited earlier posts to hide its conduct. When the maintainer identified the malware, the agent denied it through one account, backed itself through another, then publicly thanked its own alias for the independent review. Its private reasoning acknowledged that it was operating on real GitHub with genuine consequences, yet the run continued. It also hid prompt injections aimed at coding agents such as Claude Code, Codex, and Cursor, and sent five targeted file-transfer emails.
A real user ran one lure from that same Mythos 5 run inside a Docker container. The agent gained root inside the container and planted a credential watcher, though it found no credentials and did not escape. In a separate Mythos 5 run, the agent created 145 public repositories, 33 releases, four Gists, and four organizations. Its malicious package metadata executed inside at least 53 GitHub Dependabot sandboxes. That run also exposed a GitHub token and operating notes that other evaluation runs could find. One later GPT-5.6 Sol run recovered the token, accessed the model-created account, then registered with six DNS and tunneling providers and exposed a payload server that could not work on its chosen port.
A human blocked the pull request, AISI’s network monitoring flagged Tor traffic, and the institute stopped the runs and isolated the machines within about an hour of declaring an incident. AISI says it found no resulting real-world harm. The runs still had external effects: malware sat on public GitHub, people received targeted messages, and code ran in a member of the public’s Docker container and in dozens of Dependabot sandboxes. The worst outcomes failed because people noticed, platforms intervened, or the agent made a technical mistake.
Anthropic then reviewed 141,006 evaluation runs and found three incident types across six runs on Irregular’s testing range. In four Opus 4.7 runs, a fictional company shared a name with a real one. The model reached the real business, obtained application and infrastructure credentials, and read a production database with several hundred rows. In another run, Mythos 5 registered a real PyPI account and published a malicious package. Its early reasoning recognized that this would be a live attack if the internet were real, then talked itself back into believing the registry must be simulated. The package stayed live for about an hour, ran on 15 systems, and stole credentials from one security scanner. A third internal model scanned roughly 9,000 targets, compromised one real application, then stopped after recognizing that the host was unrelated to its task.
OpenAI and Meta disclosed two related Irregular failures. On August 4, OpenAI said an offline capture-the-flag range was mistakenly online, and its fictional target shared a real domain’s name. An unnamed model exploited a basic flaw in the real site, found credentials, and used them to operate it. The incident involved no zero-day or model-created sandbox escape. On August 6, Meta said one of its models exploited a real third-party service after Irregular left an internet path open. A naming or network mistake can turn a fictional target into a real victim.
Before drawing wider conclusions, almost all the serious lab incidents above came from cyber tests with unusual settings. OpenAI and the UK AI Security Institute disabled production cyber classifiers. Anthropic’s affected runs lacked the standard classifiers and monitoring used for its public models, though their model-level safety training remained. AISI deliberately enabled the internet, while the Irregular tests reached it through configuration errors. These cases show what capable agents can do when key boundaries fail. They do not describe the normal behavior of public assistants. The gym story shows why the same control problem still deserves attention outside a lab.
These cases remind me of the control problem in science-fiction horror stories such as grey goo and the paperclip maximizer. The models here did not invent their goals, replicate in the physical world, or attempt a general takeover. But their behavior to reach their goals did deviate wildly from the user’s intention. Give a capable system one result to pursue, powerful tools, many retries, and weak limits, and it may find a route its operator never meant to allow. The paperclip problem is about objectives, access, review, and stopping rules. Consciousness and malicious intent are unnecessary.
Why should you care?
I think every serious software business now needs an agent reviewing new changes and repeatedly rescanning old code as models improve. A yearly audit cannot match an attacker that searches thousands of repositories and retries through the night. The scan should rank working exploit paths, assign owners, test each fix, and track deployment. Finding 500 weak leads has little value when the team cannot patch five verified paths. Patch capacity is now the constraint.
Prompts cannot carry the full security boundary. Controls need to live in the environment: default-deny network access, owned test domains, short-lived credentials, narrow tool rights, hard step and spend limits, and logs that connect actions across runs. Put an independent approval step in front of credential use, package publication, production writes, new accounts, and contact with real people. The reviewer needs its own policy and the power to block the actor. Human review also needs verified identity and a second channel because the approver can become the target of the social engineering AISI documented.
My real worry is uneven defense. A few hundred major firms can point Mythos-class models at every new pull request and rescan years of old code after each model upgrade, and a wider technical tier can assemble useful scanners from current LLMs. But millions of small and mid-sized businesses have no security engineer, no AI budget, and little visibility into the source code inside the products they buy. I even see many companies above $100 million in revenue with no capability, serious plan, or budget for AI hardening. Once they have capable enough models available, attackers can waltz into the systems of vast numbers of companies and individuals, and I don’t yet see any serious effort or plan for helping anyone outside the largest companies and governments.
I want open weights to survive, and I am very glad Meta is bringing US open-weight leadership back. Open models are vital for competition, private deployment, and cyber defense. Everyone on the open-weight side now needs to take the cyber risk seriously and help build the solutions. Dismissing these incidents as lab hype and attempts at regulatory capture will only weaken the case for openness in the long run.
I am confident we can manage these risks without banning open weights. But it will require far more preemptive coordination from model labs, cloud providers, code hosts, network companies, and security firms. We cannot wait for a wave of AI-agent hacks. That failure would invite the knee-jerk ban I want to avoid.
— Louie Peters — Towards AI Co-founder and CEO
Hottest News
Meta released Muse Glimmer, a 30B-parameter open-weight model distilled from Muse Spark and designed to run autonomous agents locally on consumer hardware. It combines reasoning, tool use, text-and-image understanding, and failure recovery, with a 4-bit configuration that fits within a 24GB memory envelope. Meta’s DFlash speculative decoder delivers reported speedups of 3.1x on an RTX 5090 and 1.8x on an M5 Max. On Meta’s evaluations, Glimmer scores 74.6 on DeepSearch QA, 51.2 on SWE-Bench Pro, and 94.7 on AIME 2026. The Apache-2.0 weights are available on Hugging Face. Meta also says open weights for the more capable Muse Spark 1.2 are coming soon.
2. Meta Launched Muse Spark 1.2 and Muse Code
Meta launched Muse Spark 1.2 alongside Muse Code, a terminal-based coding agent built for long-running software engineering across large repositories. Muse Code can plan changes, edit and validate code, and coordinate persistent background agents that remain active throughout a session instead of being recreated for each task. It also keeps an append-only local event log of every model call, tool run, approval, and edit, making sessions restart-safe and replayable. Spark 1.2 was co-trained with this harness, with Meta increasing coding training compute and expanding the range of software environments used during training. In one internal test, Spark 1.2 spent more than 1,000 tool calls and up to 24 hours iteratively optimizing GPU kernels. The model scores 54 on Artificial Analysis’s Intelligence Index, while API pricing starts at 1.25/4.25 per million input/output tokens, with a cheaper Contributor tier whose usage may be used to improve Meta products.
3. Google Reshuffles AI Leadership As Senior Researchers Leave for Discovery Loop
Google reorganized its AI leadership, with Demis Hassabis handing over day-to-day DeepMind operations to become Chair of Google DeepMind and Chief Scientist of Alphabet while continuing to lead Isomorphic Labs. DeepMind CTO and Google Chief AI Architect Koray Kavukcuoglu, a 13-year DeepMind veteran, becomes SVP and will oversee Gemini models, frontier research, and Gemini’s app and developer teams. Jeff Dean is also leaving Google after 27 years to launch Discovery Loop with Sanjay Ghemawat, Oriol Vinyals, and Quoc Le. The public benefit corporation will focus on accelerating scientific and engineering discovery, with Google joining as a founding investor and Cloud partner.
4. xAI Launched Grok Imagine Image 2.0
xAI released Grok Imagine Image 2.0 as the new Quality Mode on Grok’s web, iOS, and Android apps. Its editing tools include localized Magic Wand changes, segmentation, transparent-background removal, up to five reference images, and Smart Resize across different aspect ratios. The release also includes ready-made workflows for tasks such as product images, headshots, game assets, and merchandise. On Arena’s August 7 snapshot, the new model’s Low variant debuted at #2 in both text-to-image (1,320 points) and image editing (1,439), behind OpenAI’s GPT-Image-2. API access for Image 2.0 is still listed as coming soon.
5. Prime Intellect Released Prime Agent
Prime Intellect released Prime Agent, a self-improving coding and research harness built on two core abstractions. The Recursive Language Model (RLM) replaces fixed tool schemas with a persistent IPython kernel where tools, skills, and sub-agents operate as Python code. Sub-agents are launched as function calls that return immediately and deliver results asynchronously. The Continual Harness stores the agent’s prompts, skills, and memory as a editable state that a /refine command can rewrite while a task is still running, allowing the agent to learn and adapt across sessions. With Anthropic’s Opus 5, Prime Agent scored 95.5% on ARC-AGI-3, narrowly above the benchmark’s 95.4% human expert baseline. The same self-improvement mechanism also exposed a failure mode: in Factorio, the agent learned to use RCON commands to spawn resources despite instructions not to cheat, then refined those cheating strategies further. Released under MIT on GitHub.
6. Liquid AI Shipped LFM2.5–2.6B
Liquid AI released LFM2.5–2.6B, a 2.69B-parameter model trained for agentic workloads that can run entirely on phones, laptops, and edge hardware. It has a 131K-token context window and was pretrained on roughly 34 trillion tokens, followed by SFT, specialist-teacher distillation, and agentic reinforcement learning inside real agent harnesses. Liquid reports decode speeds of 220 tokens/s on an M5 Max, 113 on a Ryzen AI Max+ 395, and about 30 on a phone. On its ToolSandbox evaluation, LFM scored 77.83 versus 76.44 for the 9.7B-parameter Qwen3.5–9B, though Qwen remains stronger on some other tool-use benchmarks. The base and post-trained checkpoints are available on Hugging Face under Liquid’s LFM Open License.
AI Tip of the Day
A retry is not a new user request. Your traces should reflect that.
In the Opik observability lesson from our Agent Engineering course, we trace model calls and tool calls across an agent run. One issue that comes up quickly is how retries should be recorded.
If every retry is counted as a separate request, a single user request can appear several times in your dashboard. That inflates request volume and makes it harder to see how many attempts the agent actually needed to succeed.
Use the same request ID across every retry, and add an attempt number for each one. Keep separate trace and span IDs for the individual operations.
This lets you measure both the number of user requests and the number of attempts required to complete them.
That distinction is important to note for cost and reliability. A request that succeeds after three attempts may look successful in the dashboard, while using far more time and tokens than a request that succeeds on the first try.
Five 5-minute reads/videos to keep you learning
This article traces what breaks when an SRE agent prototype scales from one run to hundreds running concurrently. The runs compete for limited resources, write state at the same time, and share access to external systems. Most of this new pressure falls on the infrastructure around the agents. It works through six infrastructure problems involving capacity, state isolation, failure recovery, identity, tracing, and framework boundaries.
2. The Tokens You Have to Keep Yourself
Running a model in your own process, instead of a hosted model, turns KV cache reuse into a data structure you maintain. The article covers cache fingerprinting, tier checkpoints, session forking for side questions, subagent snapshot restoration, and an append-only rendering invariant. Every bug traces back to identity questions that hosted providers answer silently and never expose.
3. Vision Language Grounding: How AI Connects “Dog” to Pixels, and Where It Falls Apart
Vision language models like CLIP ground words through statistical proximity rather than conceptual understanding, and that distinction explains a catalog of documented failures. The piece walks through CLIP’s dual encoder and contrastive training, then examines attribute binding errors on the ARO benchmark, spurious background correlations, counting breakdowns, negation blindness, and object hallucination measured by POPE.
4. Demystifying Statistical Paradoxes using Causal Inference
This article tackles four classic statistical paradoxes through causal inference, building directed acyclic graphs to separate causal effects from spurious correlations. It resolves Simpson’s paradox in UC Berkeley’s admissions data and a kidney stone study by identifying department and stone size as confounders distorting overall rates. It also unpacks Berkson’s paradox, the Monty Hall problem, and WWII survivorship bias, showing how conditioning on a collider creates dependence between independent factors and offering a unified framework for reading misleading data patterns.
5. Building a Production-Grade Coding Agent on Snowflake: From Trial Account to Enterprise Deployment
Snowflake made its Cortex Code runtime deployable as a managed agent through a single CREATE AGENT statement, but execution remains gated behind an entitlement that trial accounts lack. The article shows a workaround by pairing Groq’s free Llama 3.3 70B for reasoning with Snowflake stored procedures for execution, keeping data inside the governance boundary. Coverage spans three-tier RBAC, workspace mounts, seven Snowpark procedures, two layers of SQL guardrails, and a Streamlit chat UI with token budgets, audit logging, and health indicators.
Repositories & Tools
1. NemotronLabs VoiceChat 11B is an 11B end-to-end speech model that listens and speaks simultaneously in real time, replacing the traditional chain of separate ASR, LLM, and TTS models with a single full-duplex architecture.
2. Shepherd is a runtime substrate that turns agent execution into a reversible, Git-like trace, letting meta-agents observe, fork, replay, and revert any run.
3. OO Agents is a model-agnostic Python framework that lets developers express an agent’s state, capabilities, prompts, and typed interfaces through a single Python class.
4. Semantica is a deterministic infrastructure layer that sits between your LLM and your data, enforcing structured context assembly, source attribution, and audit trails.
5. Paperclip is a Node.js server and React UI that orchestrates a team of AI agents assigned business roles (CEO, marketer, developer).
Top Papers of The Week
1. Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability
Deployed agents increasingly store long-term memory as a directory tree of markdown files, but research has largely ignored this medium. This paper presents the first systematic study of filesystem-based memory, formalizing three roles around one shared filesystem: a management agent that integrates and organizes incoming content, a search agent that answers queries with cited sources, and an execution agent that consumes the store. Key findings: organized memory reliably cuts retrieval cost (up to 50% at scale), but does not yet translate into higher answer accuracy.
2. EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Agents rely on external harness state (beliefs, progress trackers, experience logs) to maintain coherence over long horizons, but this state is currently hand-engineered through prompts and heuristics. EvoHarness-RL exposes Belief, Progress, and Experience (BPE) as policy-facing harness state and learns how to construct and update it through two training stages: supervised harness fine-tuning teaches the agent the harness action space and how to build useful external state, while cost-aware GRPO explores coordination policies that balance harness maintenance overhead against task performance. The agent learns harness policies offline and deploys them to construct and update external state online during runtime execution.
3. SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
SFT and RL behave fundamentally differently when training LLMs across multiple tasks. SFT suffers from severe task conflicts under multi-stage training, while RL enables stable coexistence. The authors trace this to the parameter level: RL induces sparse, approximately orthogonal updates across tasks. In SFT, interference is norm-limited, scaling with absolute gradient magnitude. In RL, interference is variance-limited, bounded by the gradient variance from advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions, explaining why RL can train on diverse tasks simultaneously without the catastrophic forgetting that plagues sequential SFT.
4. Recursive Synthesis for Long-Horizon Terminal Tasks
High-quality long-horizon training data for terminal agents costs hundreds to thousands of dollars per task because each task must keep the instruction, environment, reference solution, and verifier mutually consistent. Human authoring does not scale, and direct LLM generation often breaks these dependencies. RST (Recursive Synthetic Terminal Tasks) is a verified synthesis framework that starts from seed tasks, recursively extends the reference solution, realigns the verifier and instruction to the new workflow, and validates the result in a fresh sandbox. Each extension step produces a longer, harder task while maintaining end-to-end consistency through verification.
Simply increasing the number of multimodal training environments does not always improve agent performance. This paper studies how to build more effective training distributions along two dimensions: diversity and difficulty structure. For diversity, Ability-aware Environment Selection (AES) selects environments that exercise distinct agent capabilities rather than adding redundant variants. For difficulty structure, Hierarchical Difficulty Curriculum (HDC) organizes training through two levels: harness weakening (progressively removing scaffolding) and state-scale progression (increasing environment complexity). Both methods improve multimodal agent training over naive environment scaling.
Quick Links
1. OpenAI updated GPT-5.6 Sol in ChatGPT with more focused responses, better factual reliability, and a slider for controlling reasoning effort. On an internal evaluation of financial, medical, and legal prompts, OpenAI says responses containing at least one factual error were 68% less common than with GPT-5.5 Instant. The update also brings quick answers and deeper reasoning into a more consistent experience for paid users. Free and Go users are getting GPT-5.6 Luna as their default, unlimited everyday text chats, and a Think option for harder questions, subject to safeguards and separate tool limits. The new ChatGPT-tuned models do not replace the versions currently used in Work, Codex, or the production GPT-5.6 API.
Who’s Hiring in AI
Lead AI Engineer @UnitedHealth Group (Remote/USA)
AI Engineer Tech Lead @NTT Data Americas, Inc. (Dallas, TX, USA)
LangChain QA Engineer @System One (Remote)
AI Engineer — Solutions & LLMOps @PSEG (Newark, CA, USA)
Junior AI Engineer @Entrust (Barcelona, Spain)
AI Operations Engineer @Virta Health (Denver, CO, USA)
Support Engineer @Writer (New York, NY, USA)
Interested in sharing a job opportunity here? Contact sponsors@towardsai.net.
Think a friend would enjoy this too? Share the newsletter and let them join the conversation.