OpenAI warns AI browser agents remain vulnerable to prompt injection

Even as OpenAI works to harden its Atlas AI browser against cyberattacks, the company admits that prompt injections, a type of attack that manipulates AI agents to follow malicious instructions often hidden in web pages or emails, is a risk that’s not going away anytime soon, raising questions about how safely AI agents can operate on the open web.

In a detailed disclosure, the ChatGPT maker said the risk has grown with the introduction of ‘agent mode’ in ChatGPT Atlas, which allows the system to view webpages and take actions such as clicks and keystrokes inside a user’s browser. “Agent mode in ChatGPT Atlas is one of the most general-purpose agentic features we’ve released to date,” the company said, noting that the agent operates in the same browser space and context as a human user.

That capability also makes the system a more attractive target. “As the browser agent helps you get more done, it also becomes a higher-value target of adversarial attacks,” OpenAI said, adding that prompt injection is “one of the most significant risks we actively defend against.”

Prompt injection attacks work by embedding hidden instructions into content an AI agent processes, such as emails, documents or websites, intending to override the user’s intent. For browser-based agents, this creates a new threat beyond traditional phishing or software exploits, because the attacker targets the agent itself rather than the human user.

The company described a hypothetical case in which a malicious email instructs an agent to forward sensitive documents to an attacker. If the user asks the agent to review emails, the system could encounter the hidden instructions and act on them. Because the agent can perform many of the same actions as a user, the potential impact could include sending emails, transferring money or altering cloud files.

It recently deployed a security update to Atlas’s browser agent after discovering a new class of prompt injection attacks through internal automated red teaming. The update includes a newly adversarially trained model and additional safeguards. “This update was prompted by a new class of prompt-injection attacks uncovered through our internal automated red teaming,” OpenAI said.

To find such attacks, the company has built an automated system that uses reinforcement learning to simulate adversarial behaviour against its own agents. The system iteratively tests and refines attacks in a simulated environment, allowing researchers to identify weaknesses before they appear outside the company. It said this approach has uncovered attack strategies not previously found by human testers or external reports.

One internal demonstration showed an agent being tricked into sending a resignation email to a company executive after encountering a malicious message planted earlier in the user’s inbox. The company said examples like this help guide improvements to its defences.

Despite these measures, the prompt injection remains an unresolved problem. “We view prompt injection as a long-term AI security challenge,” OpenAI said, comparing it to online scams that continue to evolve. The company said it expects to work on the issue “for years to come.”

OpenAI said its strategy includes continuously retraining models against newly discovered attacks, strengthening system-level safeguards and responding quickly to emerging threats. It also issued guidance to users, advising them to limit logged-in access when possible, carefully review confirmation requests and give agents narrowly defined instructions.

“Agent mode in ChatGPT Atlas is powerful, and it also expands the security threat surface,” the company said. “Being clear-eyed about that tradeoff is part of building responsibly.”