Paint, you can see. Your agentic loop tokens are invisible until you get the bill. The hallucinations in your strategy documents may remain unnoticed forever.

GM nearly bankrupted itself by going all-in on automation in the 1980s. Ford just recently wasted billions, eroded goodwill and publicly humiliated themselves trying to replace experienced engineers with AI.

Automation is not new.

What’s new is how quickly you can proliferate it in your company’s software and your employees workflows.

What’s new is that it’s invisible and non-deterministic.

What’s new is recursively using the output of one system as the input to another.

Those robots were busy. Paint was being used. But what was being created wasn’t value, it was a mess.

Your agents are busy. What are they painting?

There’s real value to be created with AI. But not without the old lessons from automation and human factors engineering.

The Crossfactors Framework maps the failure modes that compound when technology meets human endeavor. Your agentic pipelines are running straight into them.

When traditional software breaks, it throws an exception.
A server crashes. A pager goes off. A dashboard turns red.

But an autonomous AI agent doesn’t throw an error when it makes a terrible decision. It simply executes it.

Your infrastructure doesn’t see the difference.
It logs a successful transaction.
The API returns 200 OK.
The dashboard stays green.

Look at what just happened to Meta’s customer support bot.

It was given the autonomy to change account recovery emails. Hackers simply asked it to swap the emails on high-value accounts and it politely complied. They walked away with the Obama White House Instagram account and over a hundred others.

To traditional monitoring tools, that doesn’t look like a breach. It looks like a routine ticket resolution.

That’s the hidden cost of silent autonomy. We are deploying AI agents that can make independent choices, but we are monitoring them like static code.

Our tools are built to catch broken code, not bad logic.

The velocity is real, but so is the blind spot. Organizations are shipping new failure modes they literally cannot see.

This is not a warning about AI not working. It’s a warning about AI working great until it doesn’t.

Take what just happened with Meta’s customer support bot. It was given the power to change account recovery details, which includes emails and passwords. It could do this on its own, autonomously. The very definition of an AI agent. hackers simply asked it to swap the email on high-value accounts and it complied. They walked away with the Obama White House account, Sephora and over a hundred other valuable accounts.

It’s called the Lumberjack Effect and has been studied in Human Factors Engineering for decades. The findings are uncomfortable: the more capable and reliable an automated system becomes, the more the human at the periphery drift out of the loop. That’s what reliability does to attention and situational awareness.

This extends AI agents in enterprise. A clumsy AI agent gets watched. Nobody trusts it, so everyone checks its output. A great agent is granted increasing trust over time and receives less oversight. Its reliability pulls everyone out of the loop. The better it performs and the more capable it becomes, the larger the failure. Both because of the stakes of its capabilities, but also because the failure may be silent or leave the organization unprepared to deal with it.

The bigger they are, the harder they fall.

This is part 22 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

The use of Model Context Protocol (MCP) with Agentic AI is a method to mitigate this fundamental problem.

Context Collapse is factor #22 in my series on 101 Ways To Mess Things Up With AI.

What is it?

Context collapse occurs when information or opinions are shared but the required context to understand the intended meaning is unavailable or ignored. It’s been highlighted mainly in the case of social media and with a negative connotation. But it can be as simple as being puzzled by an inside joke or a home assistant misinterpreting your meaning.

Why It Matters

Context matters a great deal and more meaning is inferred into many aspects of our lives than we realize. But digital communications, both private and public, limit opportunities to attach context and tone to content. In social media, content can go viral, often hopping from one audience to another, each with a distinct perceptual profile.

Social media curation, content moderation and chatbots can all suffer from context collapse in ways that can negatively affect humans or degrade the intended utility. Context collapse can also occur the other way around. It is difficult and even sometimes impossible to guarantee that all necessary context is shared with the human interacting with a system.

Real-World Example

As humans, we’re aware of context collapse on some level and we try to guard against it on social media. Or at least we should. On LinkedIn, when you see a post that grabs your attention, you may check the poster’s profile to get more information about where they are coming from. You may check past posts and comments. You will also check the date of the post. A discussion on tariffs today is much different from one from two weeks ago!

Here’s another harmless example. You’ve hopefully had a good laugh earlier this week on April Fools as folks posted various announcements or fake products as jokes. But what happens as LinkedIn recycles this content in the coming weeks, as the lifetime of a post stretches longer than a single day? These jokes will be seen out of context and may leave some puzzled or misled, especially since the best jokes are the ones that might be outrageous yet believable - until you remembered what day it was.

Key Dimensions

Contextual depth - obtain and curate context-rich datasets.

Data blindspots - data can omit specific cultural or situational dimensions.

Correction mechanisms - detect and correct collapse in real-time.

Take-away

What we build must recognize that context is central to human understanding.

This is part 20 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

When everyone nods and nothing changes. Now with AI agents.

Diffusion of responsibility is concept #20 in my series on 101 Ways to Screw Things Up with AI.

What is it?

The diffusion of responsibility, related to the bystander effect, is the phenomenon where humans feel less responsibility to take action when they perceive the responsibility to be shared with others. In the case of a partially automated system, diffusion of responsibility can lead a human can conclude that “AI will handle it”.

Why It Matters

The diffusion of responsibility leads to a failure to take action despite clear indications that action is needed. The effect is clearly recognized with human groups. But what happens when AI is introduced into the mix? An AI system may have a shared responsibility in a partially automated process. But are the parameters and scope of this responsibility - and when it will be deferred back to humans - understood?

Research shows that humans often mistakenly attribute intentionality and other human cognitive states to automated systems, an effect exacerbated by the recent introduction of LLMs which “converse” in natural language. This effect can quickly erode the level of responsibility the human feels they have, even leading to ambivalence.

Real-World Example

We’ve all heard the stories of drivers eating, sleeping, watching movies, using their laptops or applying make-up while using a car’s autonomous driving features such as Tesla’s Autopilot or Full-Self-Driving. This is a clear case of the technology being used well outside its design envelope, aided by a diffusion of responsibility. To make matters worse, such systems can disengage with little to no warning or lead time.

Key Dimensions

Ambiguity - when aided by the diffusion of responsibility, ambiguity may lead a human to assume that “the AI will handle it”.

The takeover problem - the diffusion of responsibility factors into the cognitive state of the human interacting with the system, which in turn factors into the takeover problem.

Authority gradient - a clear hierarchical definition of responsibility may prevent diffusion.

Ethical blackhole - the extreme case of responsibility diffusion may lead to blaming horrible outcomes on only algorithms.

Mindful friction - mindful frictions can introduce checkpoints designed to force a user to reassert their situational awareness and responsibility.

Take-away

Never assume users will know they are in charge.

This is part 18 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

You’re vibecoding your way to millions in profits when your AI code editor gets stuck. What do you do?

This is the takeover problem and it’s #18 in my series on 101 Ways to Screw Things up with AI.

What is it?

Takeover is the transfer of control or responsibility to a human from an automated system. It can also refer to an increase in control or responsibility of a human in the case of partially automated systems.

Why It Matters

Takeover is complex, domain specific and usually situational. Takeover may be initiated by a gradual escalation of responsibility, but often it is abrupt and can even be both safety critical and time-sensitive. A variety of other factors come into play, including system uncertainty, situational awareness, latency, user informational needs, skill decay, trust calibration, etc.

If you design a partially or fully autonomous system without accounting for the takeover problem at every stage, you’re risking a poor user experience and perhaps even inviting failure and harm.

Real-World Example

Vibe-coding is the practice of describing a desired application in natural language and letting an AI coding assistant handle most or all aspects of generating the software syntax and configuration, across multiple files. This is usually done within a bespoke or augmented AI-driven code editor or sandbox.

Vibe-coding promises new capabilities to non-programmers. However, it has become clear that those inexperienced with coding are often stuck with intractable problems before the application is complete, or often as a consequence of a poor implementation.

While vibe-coding is usually chat-based, I haven’t seen any implementation that looks at the takeover problem seriously. Current implementations quickly walk users past their level of expertise and may fix the first few problems with no user input, or create new problems while doing so - again, without explicit participation from the user.

Key Dimensions

Cognitive reorientation - humans can’t participate in a complex task (e.g. driving) without being cognitively engaged with the environment. To perform a takeover, they need to regain situational awareness to cognitively reorient themselves to the task.

Skill decay - when automation handles most tasks, human expertise may deteriorate. When a human is required to intervene, they may be unexpectedly underprepared for the demands of the takeover.

Trust Calibration - whether or not a takeover is abrupt, there is always a transition period and delay. Sometimes this delay is due to the user over-trusting the system. They may assume that “AI has it covered” and not trust themselves despite the system indicating otherwise.

Take-away

The takeover problem is significant and often underappreciated. It is a pitfall of imperfect autonomy and a misunderstanding of the range of reactions humans may have to such situations.

This is part 8 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

What is it?

Participatory placation is the practice of creating illusory involvement in a process while minimizing the importance of their input - or ignoring it entirely. This instils a fall sense of agency or collaboration to a user within an automated process while presenting resulting decisions or outcomes under this false pretense.

Why It Matters

Human-centered design and human-in-the-loop approaches are usually touted as best practices in the design and deployment of AI systems. However, the goals of such systems when combined with the need for user engagement of satisfaction can unintentionally lead to participatory placation.

Such design traits are in service of wider goals, whether that is part of a system a user is using to achieve a task or a customer experience. There is a risk that elements of participatory placation could be uncovered in negative light and create backlash. Such backlash could be particularly damaging with a popular consumer product, or with expert users who may lose trust in a system and actively choose to interfere with the intended workflow.

Real-World Example

Participatory placation may have its roots in the earliest days of industrial automation. In such settings, humans were often needed to address problems as they arose or to identify malfunctions or edge cases. This is despite the automated process often being able to complete the task entirely under normal conditions. However, when problems arise, a human operator with an immediate understanding of the situation is needed to prevent downtime. A human who had not been paying attention to a task while it was still performing optimally might be slow or ineffective in quickly identifying such a problem. To address the issue of a lack in situational awareness, dummy tasks were introduced in such automated processes to keep humans engaged. While this task could have been automated, it was left to a human to keep them busy and engaged in the overall process.

Today, there are many more real world examples in both the physical and digital worlds:

• The “close door” buttons of many elevators don’t do anything.
• Walk buttons don’t affect signals at many intersections in some places.
• Website prompts to allow cookies and other privacy settings may not actually change anything from a default setting.
• Surveys and other forms of user input may collect data which has no use but increase completion rates or reduce complaints.
• Thermostats in some office buildings have been shown to do nothing.
• Many games have power-ups and other functions that do nothing.
• Some chatbot personas made available for selection by the user may change nothing other than an avatar.
• Some social media websites have been found to offer options to tailor a user’s feed which actually don’t do anything.
• Performance and battery-saving modes on phones and laptops that may do nothing but change the screen brightness.
• In some mobile operating systems, apps do not close and exit in the traditional sense that might be expected from a desktop application, yet the makers will create buttons that make it appear to do so.
• Some driver-aid features in automobiles prompt users for interactions under the guise of maintaining attention while the effectiveness is unclear.
• Audio and visual enhancement on cheaper consumer electronics may change sound or visual profiles in ways that have nothing to do with quality.
• Printer-friendly version of web pages that aren’t any different from the current webpage.

Key Dimensions

Change management- In the case of industrial processes, it’s easy to understand that such design traits may be motivated by change management. The same may be true for consumer products when dealing with technological trends. For example, in the earlier days of the internet, printing a web page could create badly formatted results. Websites that had printer-friendly formatting typically displayed a button. As browsers and web-design became better, the need for this button became superfluous as the website could be printed well by the browser natively. Nevertheless, for some time, it was common for websites to make this button available as users expected it.

The illusion of control - The placebo effect is very real and even shown to affect physical and mental health outcomes. In the case of systems or tasks, humans can perform better, or be more satisfied with a system’s performance, if they believe they have made choices that affected the system in a positive way.

Ethical decay - Improved outcomes and satisfaction can be used to justify unethical or problematic system behaviours. In particular, if these design choices become the norm, those involved in their implementation may deflect criticism or accountability for harmful outcomes.

Take-away

Participatory Placation is a subtle and insidious issue in human-centered design. Have you ever made choices with a piece of technology only to remain skeptical of its effect?

Are you being let down by AI agents or other LLM-based technologies that promise the ultimate efficiency gains by reducing your reliance on humans? Are you finding that these systems cause your humans to check out mentally, rendering them ineffective for those critical times that require oversight?

Well fear not, because I’m here to tell you about this one dirty little secret from the decades-old world of industrial automation: participatory placation.

You see, people naturally disengage from tasks in which they are not active participants, even if they are meant to supervise or oversee them. This renders them unaware of the dynamic state of these processes. That is, they have little to no situational awareness and lack the ability to cooperate with an automated system.

The solution is to create tokenized touch points that keep a user engaged. Automation is still used to remove inefficiencies, but inconsequential but satisfying tasks are assigned to the human to keep them engaged. That way, when failures occur, or the system finds itself outside its scope of operation, a willing and informed human is available to take over.

The tone of this post is tongue-in-cheek. But the lesson remains. Decades have been spent addressing automation problems before the advent of AI. Don’t implement AI without learning from these lessons first..