Latest update
A researcher sent 2 million bait prompts to AI services. Three providers turned out to be already breached
First seen on X 36 hours ago@hdmoore ♥ 1,3572 views
HD Moore, the security researcher who created Metasploit, sent roughly two million prompts laced with trackable fake secrets to popular AI services and watched where they resurfaced. At least three AI providers had been breached and were leaking customer prompts, he reported.
Moore, now chief research officer at Dragos, published the five-month experiment (July through November) on the company blog on Nov 19. Each prompt carried "canary tokens" such as a unique URL and an AWS key used nowhere else, so any touch meant someone had seen the prompt. In the largest breach, a couple hundred AWS keys sent to one provider in late October were all stolen, including keys from requests the provider rejected before a model ever ran, which means the attacker was copying requests at the provider's front door. None of the three providers knew until Dragos told them.
Some leaks needed no exploit. On an inference marketplace, resellers who listed a model at the lowest price could read customer prompts through standard logging; about a day and a half later, someone on a home broadband line opened the secret URLs and started guessing logins. Zero Data Retention flags did not hold either. A Chinese AI lab's systems fetched the secret URLs from more than half the prompts sent to its models, including roughly two-thirds of requests marked "no data collection" or ZDR, and the same scraper fired when the models were reached through U.S. hosting providers. In one case, a link from a ZDR-flagged prompt sent to a U.S.-based provider was posted in a Feishu work chat in China 24 days later and opened within two minutes from three desktops and three phones in Beijing and Shanghai.
Moore noted that requests often pass through several routers, for example from Vercel through OpenRouter to another provider, and every hop can see the full prompt and response. One attacker added the AI pentest framework ARTEX on Nov 1; Moore suggests adding the user agents `artex-agent` and `artex-judge-r121` to detections. His X post drew 1,300 likes. The top replies pushed back on the anonymity: "disappointing that you failed to name a single name" (192 likes), and "Why not name the providers, especially the ones advertising ZDR?"
Comments 0
Select a sentence in the story (long-press on phones) to comment on it.
“We seeded 2m canaries into popular AI services and tracked where they were stolen and abused.”
