The best AI-agents fall the hardest.
This is not a warning about AI not working. It’s a warning about AI working great until it doesn’t.
Take what just happened with Meta’s customer support bot. It was given the power to change account recovery details, which includes emails and passwords. It could do this on its own, autonomously. The very definition of an AI agent. hackers simply asked it to swap the email on high-value accounts and it complied. They walked away with the Obama White House account, Sephora and over a hundred other valuable accounts.
It’s called the Lumberjack Effect and has been studied in Human Factors Engineering for decades. The findings are uncomfortable: the more capable and reliable an automated system becomes, the more the human at the periphery drift out of the loop. That’s what reliability does to attention and situational awareness.
This extends AI agents in enterprise. A clumsy AI agent gets watched. Nobody trusts it, so everyone checks its output. A great agent is granted increasing trust over time and receives less oversight. Its reliability pulls everyone out of the loop. The better it performs and the more capable it becomes, the larger the failure. Both because of the stakes of its capabilities, but also because the failure may be silent or leave the organization unprepared to deal with it.
The bigger they are, the harder they fall.