Firm apologizes after AI help agent invents coverage that causes consumer uproar
On Monday, a developer utilizing the favored AI-powered code editor Cursor observed one thing unusual: Switching between machines immediately logged them out, breaking a standard workflow for programmers who use a number of gadgets. When the consumer contacted Cursor help, an agent named “Sam” advised them it was anticipated conduct beneath a brand new coverage. However no such coverage existed, and Sam was a bot. The AI mannequin made the coverage up, sparking a wave of complaints and cancellation threats documented on Hacker Information and Reddit.
This marks the newest occasion of AI confabulations (additionally referred to as “hallucinations”) inflicting potential enterprise injury. Confabulations are a sort of “artistic gap-filling” response the place AI fashions invent plausible-sounding however false info. As an alternative of admitting uncertainty, AI fashions usually prioritize creating believable, assured responses, even when which means manufacturing info from scratch.
For firms deploying these techniques in customer-facing roles with out human oversight, the results might be instant and expensive: annoyed prospects, broken belief, and, in Cursor’s case, doubtlessly canceled subscriptions.
The way it unfolded
The incident started when a Reddit consumer named BrokenToasterOven observed that whereas swapping between a desktop, laptop computer, and a distant dev field, Cursor periods had been unexpectedly terminated.
“Logging into Cursor on one machine instantly invalidates the session on every other machine,” BrokenToasterOven wrote in a message that was later deleted by r/cursor moderators. “It is a important UX regression.”
Confused and annoyed, the consumer wrote an e-mail to Cursor help and shortly acquired a reply from Sam: “Cursor is designed to work with one system per subscription as a core safety function,” learn the e-mail reply. The response sounded definitive and official, and the consumer didn’t suspect that Sam was not human.
After the preliminary Reddit submit, customers took the submit as official affirmation of an precise coverage change—one which broke habits important to many programmers’ day by day routines. “Multi-device workflows are desk stakes for devs,” wrote one consumer.
Shortly afterward, a number of customers publicly introduced their subscription cancellations on Reddit, citing the non-existent coverage as their cause. “I actually simply cancelled my sub,” wrote the unique Reddit poster, including that their office was now “purging it utterly.” Others joined in: “Yep, I am canceling as properly, that is asinine.” Quickly after, moderators locked the Reddit thread and eliminated the unique submit.
