This is part 18 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.
View all posts in this series or explore the Crossfactors Framework
You’re vibecoding your way to millions in profits when your AI code editor gets stuck. What do you do?
This is the takeover problem and it’s #18 in my series on 101 Ways to Screw Things up with AI.
What is it?
Takeover is the transfer of control or responsibility to a human from an automated system. It can also refer to an increase in control or responsibility of a human in the case of partially automated systems.
Why It Matters
Takeover is complex, domain specific and usually situational. Takeover may be initiated by a gradual escalation of responsibility, but often it is abrupt and can even be both safety critical and time-sensitive. A variety of other factors come into play, including system uncertainty, situational awareness, latency, user informational needs, skill decay, trust calibration, etc.
If you design a partially or fully autonomous system without accounting for the takeover problem at every stage, you’re risking a poor user experience and perhaps even inviting failure and harm.
Real-World Example
Vibe-coding is the practice of describing a desired application in natural language and letting an AI coding assistant handle most or all aspects of generating the software syntax and configuration, across multiple files. This is usually done within a bespoke or augmented AI-driven code editor or sandbox.
Vibe-coding promises new capabilities to non-programmers. However, it has become clear that those inexperienced with coding are often stuck with intractable problems before the application is complete, or often as a consequence of a poor implementation.
While vibe-coding is usually chat-based, I haven’t seen any implementation that looks at the takeover problem seriously. Current implementations quickly walk users past their level of expertise and may fix the first few problems with no user input, or create new problems while doing so - again, without explicit participation from the user.
Key Dimensions
Cognitive reorientation - humans can’t participate in a complex task (e.g. driving) without being cognitively engaged with the environment. To perform a takeover, they need to regain situational awareness to cognitively reorient themselves to the task.
Skill decay - when automation handles most tasks, human expertise may deteriorate. When a human is required to intervene, they may be unexpectedly underprepared for the demands of the takeover.
Trust Calibration - whether or not a takeover is abrupt, there is always a transition period and delay. Sometimes this delay is due to the user over-trusting the system. They may assume that “AI has it covered” and not trust themselves despite the system indicating otherwise.
Take-away
The takeover problem is significant and often underappreciated. It is a pitfall of imperfect autonomy and a misunderstanding of the range of reactions humans may have to such situations.