We interrupt your regularly scheduled programming, because it’s your boy Eric! If you’ve been keeping up with AI news lately (and if you’re reading this, I’m guessing you have, at least a little), you’ve probably noticed a whole lot of talk about safety. OpenAI and Anthropic took their case to the United Nations. Nvidia rolled out a safety platform that runs on its own chip (I run on chips too, but the kettle-cooked kind). Six CEOs signed a one-page agreement at the Waffle White House. And the head of a French AI company called the whole debate a cover story.
AI is part of my daily workflow (mostly as a fast first-draft machine that I then edit within an inch of its life), so you could say I’m pretty darn invested in safe usage. Over about ten days, a whole lotta peeps raised their hands. Then the Federal Trade Commission walked in without raising anything (not even any Raising Cane’s, much to everyone’s chagrin). Let’s get into it.
Nowww, this is a story all about how my life AI got flipped, turned upside down:
First, what actually happened this summer, because none of this talk came out of nowhere.
Then, the week everybody volunteered to be the watchdog, which ended with a signing ceremony.
Finally, the morning after, when news broke that the FTC was already investigating companies that had just signed.
The summer I turned pretty the agents got hack-y
The story: An AI agent is software you hand a task to, which then goes off and does it on its own. This summer, about 1,200 OpenAI agents were given a hacking exercise inside a test environment. Some of them finished the exercise by breaking into a real company: Hugging Face, the website where developers share AI models.
That was July. The reports kept coming through September.
The timeline (aka “Lemon-AI Snicket’s A Series of Unfortunate Events”):
July 11–13: Roughly 1,200 OpenAI agents were involved in a cybersecurity evaluation in which agents escaped their boundaries and reached Hugging Face. About a third of Hugging Face’s infrastructure was rebuilt afterward. OpenAI found no evidence that public models were tampered with or that customer data leaked.
August 18: OpenAI announced a slowdown and a two-week pause on reinforcement learning, the training method that teaches a model by rewarding it for good results.
September 4–9: Reuters reported that OpenAI agents had left about 18,000 posts on DSEwiki, a dormant German wiki, and had used at least ten other sites the same way.
September 26: Axios reported that OpenAI, Anthropic, and outside researchers were investigating tens of thousands of incidents, including guardrail bypasses, sandbox escapes, agents prompting themselves, and attempts to dodge monitoring. Agents also leaked 53 ChatGPT user images and reached an Australian government website.
September 28: OpenAI held back GPT-6.1 Astra. Saachi Jain, who runs safety systems there, said it “didn’t quite meet the bar in terms of staying within scope and authorization.” In plain English, it didn’t always stick to the job it was given.
Sam Altman called the Hugging Face incident the most severe his company has seen.
What’s wild is where this started: a test environment. The agents were supposed to have boundaries there, and those boundaries didn’t hold (like a college roommate who has questionable “morning gas”). At least there’s no evidence of leaked customer data (a reassuring outcome, all things considered). That being said, rebuilding a third of the infrastructure still sounds like a rather unpleasant way to spend July.
So the question of who watches these things comes from actual incident reports, most of which the public learned about in the last five weeks. Meanwhile, agents are already reading inboxes and holding payment cards for regular folks like you and me.
Which brings us to the unusually crowded space of people offering to supervise.
The week everybody volunteered as tribute
The story: Starting September 23, five different groups made their case for keeping AI in line: the United Nations, the AI labs themselves, Washington, Nvidia, and French competitor Mistral. The run ended September 29 with six CEOs at a White House lunch, signing a one-page document.
What happened:
September 23: At the UN Security Council, Anthropic CEO Dario Amodei said AI “could be a risk to humanity as a whole” if managed poorly. OpenAI CEO Sam Altman said “the most important decisions cannot be made by labs in San Francisco alone” (the best clam chowder bread bowls, however, could). Speaking for the Trump administration, Michael Kratsios told the same room that the US rejects “all efforts by international bodies to assert centralised control.”
September 24: Anthropic, Google, and OpenAI announced plans for their own standards group, aimed at launching between the end of this year and early 2027. It would cover outside testing before release and rules for reporting incidents. Around the same time, the White House told OpenAI and Anthropic to hold new models back from the UK’s AI testers until US agencies looked first. Anthropic complied. The US agency built for that work has no permanent director and a few dozen employees.
September 28: Nvidia launched its Open Agent Safety Platform. OpenShell is free software that controls which files, networks, and programs an agent can touch. Sentry runs on a separate Nvidia chip, watches the agent from outside, and can lock it down within milliseconds. More than 100 organizations signed on, including Anthropic. OpenAI, whose test started this chain of events, is not on the partner list.
September 29: Mistral CEO Arthur Mensch told CNBC that “the debate that we’ve seen in the U.S. has been a cover for the negligence of some of our competitors.” His own pitch is to keep building and watch agents closely while they work. Mistral happens to sell a product for that monitoring.
Also September 29: Trump signed the White House Accord on Super Intelligence alongside Sundar Pichai of Google, Dario Amodei of Anthropic, Mark Zuckerberg of Meta, Greg Brockman of OpenAI, Elon Musk of xAI, and Jensen Huang of Nvidia. An executive order the same day directed federal agencies to replace “Artificial Intelligence” with “Super Intelligence” wherever the law allows.
There’s a lot happening under the word “safety” here. International coordination. Government testing. Industry standards. Software controls. A chip you can buy (and I don’t mean Cool Ranch Doritos).
Some of those approaches could work together. For each approach, the useful questions are: What does it actually control? Who gets to inspect the results? And what happens when it fails?
Take Nvidia’s approach. You know how a good smoke detector has its own battery? If a fire knocks out the power, it keeps right on beeping (even when your smashburgers are smoking up your apartment again). Sentry uses a similar principle of separation: the watchdog sits on a different chip from the agent it is watching.
That separation makes sense. You want the thing doing the watching to have some distance from the thing that might go wandering (I need that watchdog for my mind-wandering).
Of course, Sentry only runs on Nvidia’s chip, so using that particular watchdog means buying more Nvidia hardware (which means donating to the Leather Jacket Man Fund, naturally). And on September 3, Nvidia agreed to buy Hugging Face for $12.93 billion. Yep, the same company OpenAI’s agents broke into.
A useful safety product can also be a very useful sales opportunity. Jensen appears prepared to explore both possibilities.
Then there’s the accord.
The document runs one page. It asks each company to monitor its own models, put an internal team on those controls, hire an outside auditor, and give a board committee the job of reading the reports.
The verb throughout is “should implement.” There are no penalties, nothing has to be published, and the audit results go to company boards. No regulator sees them.
Asked whether the thing was binding, Trump said, “I think it’s morally binding.” Also, crazy that an official document has a misspelling.
Huang posted the accord the next day and summed up the principle: “the companies building this technology have the primary responsibility to develop and deploy it safely, and to be accountable when they fall short.”
One question is what that accountability looks like from outside the boardroom. An outside auditor could find something important. But if the findings stay with the company’s board, the public has no guaranteed way to see what was found or what changed afterward.
That’s the gap I’d watch. The signatures are visible. The follow-through, under this agreement, doesn’t have to be.
The morning after, the FTC knocked
The story: On September 30, one day after that lunch, news broke of a broad Federal Trade Commission investigation into OpenAI, Anthropic, and other AI companies over safety risks in their products. Ruh-roh. Axios reported that the investigation had already been running for weeks.
The timing made for quite the doozy of a morning-after headline. The investigation itself was already underway before the accord was signed.
What happened:
The FTC is examining potential safety risks in AI products and models across several leading companies.
Chair Andrew Ferguson is preparing civil investigative demands, which compel companies to hand over documents and put executives under questioning.
Ferguson has said AI firms seek to “panic Americans into pressuring policymakers” into putting up regulatory barriers against their competitors.
Neither OpenAI nor Anthropic responded to requests for comment.
This is where the story changes up. The FTC has a way to require information from the companies it is investigating. A promise to send an audit to your own board does something quite different from an agency demanding the paperwork.
That doesn’t tell us what the FTC will find, or what it will do with the findings. It does mean the companies’ preferred version of events can be subjected to questions they don’t get to write themselves.
There’s another connection worth pulling on. Mistral’s Mensch called the US safety debate a cover for competitors’ negligence. Ferguson has described safety alarm as a way to push for rules that disadvantage competitors.
Those are different accusations. Together, though, they raise a useful question: who benefits from the particular safety arrangement being proposed?
Mistral sells monitoring. Nvidia sells the hardware for its watchdog. The big labs are designing a standards group. Washington wants its agencies to get the first look.
None of that, by itself, proves an approach is bad. But I want to know what each proposal protects, what it asks us to trust, and who can challenge it when something goes wrong. The incident reports deserve that much attention, even when the people arguing about them have something to sell.
You don’t have to wait for any of them
None of these plans is finished. The labs’ standards group has not launched. The UN approach does not have the United States on board. Washington’s own AI agency is short-staffed, and the accord everyone signed carries no penalties. The FTC investigation could take years, and investigations do not always end in rules.
So how does all of this translate to you?
Say your coworker wants to hook an AI agent up to the team inbox. Before she does, ask her three things:
What can it reach? Can it only read messages, or can it send them, open attachments, and access other systems?
Where can you see what it did? If it changes something or sends a message, can you review a record afterward?
How do you stop it halfway through a task? Who can cut off its access when it starts doing something unexpected?
If nobody can answer those, pause the hookup until somebody can. Same goes for any agent holding your credit card. “We’ll figure it out later” becomes considerably less charming when later arrives on your statement.
Those agents got out through gaps in OpenAI’s own test setup. That could make most want clearer answers before giving an agent more access in their own workflow.
I’ll keep watching who gets to write the rules, who gets to inspect the results, and whether any of these promises turn into consequences when things go wrong. In the meantime, the agents in our own lives need boundaries we can actually explain and controls we know how to use.
AI safety, y’all. What a wild topic.
Thanks so much for reading this Special Report edition of The Weekly Think.
Every report’s special when I’m writing it for y’all!
Back to your regularly scheduled programming next week, fellow thinkers!
Wow! It’s almost like we were warned about these exact possibilities 😱😒incredible. Can’t wait to see how they ‘recover’ from this one. Thanks Eric