It’s not just a feeling. This has been studied extensively across industries and work functions. Humans are not robots.

For decades, automation technologies have been introduced to help no only with efficiency, but also to reduce human mistakes.

But many ironies surface when we look closely at how humans interact with automation technologies.

One of these ironies is that new types of mistakes/errors/effects arise in places only made possible by the use of automation.

The research is clear on this too. Human errors in automated settings have patterns that depend on the time of day, the point in the shift and the day of the week.

What does this mean for AI in the enterprise. There’s no reason to think the same isn’t true. But it’s all domain and function specific.

I can turn that feeling into data. And then into action. The best human-in-the-loop workflows account for the strength of the automation in light of the limitatons of the human.

Automating repetitive work turns the expert into a monitor. Monitoring is a different job and people are bad at it.

Aviation learned this (and redesigned how pilots and autopilots work together).

Self-driving relearned it (after promising 1M robotaxis by 2020).

Radiology relearned it (after insisting that we stop training radiologists in 2016).

Now every enterprise is relearning it.

Karpathy said it himself this week: “human review and comprehension are starting to fall behind.”

It has been for a while. Give people tasks that are repetitive or demanding, and they turn into rubber stamps. Not because they’re lazy or incompetent. Because that’s how human attention works.

We’ve had forty years to learn how to design for the person watching the machine. Most AI features still ship with an Approve button and a disclaimer.

What share of your AI’s output is approved without changes? That’s your rubber-stamp rate.

You’ve probably heard this fun fact: giant ground sloths were responsible for spreading avocados because they were able to eat the pits and later disperse them miles away - something smaller mammals couldn’t do.

Just one problem. This is likely untrue!

This “fact” began its life as a speculative mention in a journal that slipped past peer review. But since then, it has been cited and mentioned so many times that the frequency alone has become an indicator of truth. It’s repeated in articles, podcasts and even by scientists. The very excellent educational YouTube channel SciShow recently published a correction episode explaining this mistake.

Right now, your engineering team is celebrating because they just plugged an LLM into your corporate knowledge base.

But if you work in AI Governance or Risk, you know your SharePoint is full of sloths: deprecated policies, draft memos from decades past and unvetted Slack opinions.

Here’s why that’s a problem, as outlined in the Crossfactor Atlas

1. Unearned Warrant - A Monday morning brain-fart from a 3-year-old project channel gets retrieved by your AI. Because it is being served up by an “intelligent” system, without any of the original context, it is granted unearned warrant. It is given the same corporate weight as formal, reviewed documents the system also queries.

2. Surface Credibility - Because the AI outputs obsolete information with perfect grammar and authoritative formatting, your employees blindly trust it. The cleanliness of the output hides the messy nature of the input.

3. Authority Capture - The AI’s output gets included into a new reference document. The document gets included into a new SOP. The dirty AI output has now accrued the ultimate trust and citational weight faster than your Governance team can establish accountability.

Your LLM doesn’t need to hallucinate to poison your institutional knowledge.

Does your team have the vocabulary to find these hidden failure modes?

(P.S. I map out over 200 of these exact sociotechnical risks in the Crossfactor Atlas. DM me if you want a copy).

Claude’s watermarking is not here to free LinkedIn of AI slop or rat you out when it authors an invitation for you.

Watermarking is here to help Claude gain adoption in enterprise.

Enterprise Risk and AI Governance leaders know what’s happening. They are terrified their internal projects are quietly filling up with AI slop instead of actual human output.

They are hoping a watermark will finally let them see who is doing the work. But this problem runs much deeper than it seems. Using technical features to solve human problems is shortsighted.

Here’s the risk profile that’s at play, as laid out in my Crossfactor Atlas.

1. Workslop. You’ve heard of this. When employees use Claude to write a 6-page memo which will then be summarized by the recipients using the same chatbot.

2. Effort Laundering. A watermark is a hurdle, not a wall. If employees are incentivized to ship volume over quality, they will simply run watermarked text through secondary, open-source tools to strip the cryptographic signal.

3. Institutional amnesia. If you are relying on watermarks to catch slop, you are already losing. By the time you catch it, your team has been substituting AI for experience for months.

4. Surface Credibility. Trust is granted to content based on its presentation (dashboard polish, memo structure, graphics, illustrations etc) before its substance is even assessed.

5. Trust Transference. Once your workforce expects AI output to be watermarked, unmarked material may gain credibility it does not deserve.

6. Unearned Warrant. Content entering your organizations shared record can quickly become disconnected from the initial verification. Information can be re-cited until each hop reads as independent corroboration.

7. Authority Capture. Related but distinct to many of the above. AI output can accrue trust, standing and citational weight within an organization faster than accountability for those outputs is established.

Enterprise leaders are hoping a watermark will act as a lie detector for human effort. But watermarking is an IT solution to a sociotechnical problem. It will fall well short of solving workslop and the other risks listed above.

These are just 7 of the 212 risks I map out in the Crossfactors Atlas.

AI Governance and Risk leads out there: how many of these are you actually tracking today?

Congratulations on your new AI rollout. Your team is now quickly becoming complacent instead of using their judgment.

Which of these AI solutions will bring you brand-new problems?

1. Prompt engineering.
2. Human-in-the-loop.
3. RAG.
4. API safety guardrails.
5. Model evals.
6. Red teaming.
7. Risk frameworks.
8. Dashboards.
9. soTA models.
10. Agents.
11. AI upskilling.
12. Copilots.
13. Context engineering.
14. Governance Committees.
15. Fine-tuning.

Don’t get me wrong; these can all be valuable.
But they all have problems.

Leave a comment if you disagree or think I missed something.

Check out the free 5-minute quiz on my profile to see where your team stands.

Follow me to learn why AI creates new problems for teams.

Paint, you can see. Your agentic loop tokens are invisible until you get the bill. The hallucinations in your strategy documents may remain unnoticed forever.

GM nearly bankrupted itself by going all-in on automation in the 1980s. Ford just recently wasted billions, eroded goodwill and publicly humiliated themselves trying to replace experienced engineers with AI.

Automation is not new.

What’s new is how quickly you can proliferate it in your company’s software and your employees workflows.

What’s new is that it’s invisible and non-deterministic.

What’s new is recursively using the output of one system as the input to another.

Those robots were busy. Paint was being used. But what was being created wasn’t value, it was a mess.

Your agents are busy. What are they painting?

There’s real value to be created with AI. But not without the old lessons from automation and human factors engineering.

The Crossfactors Framework maps the failure modes that compound when technology meets human endeavor. Your agentic pipelines are running straight into them.

That’s what Ford just did. They should have learned from GM.

In the 1980s, GM poured $45 billion into factory automation in an effort to catch up with Toyota. Then CEO Roger Smith said “Automation came along just in time to save us”.

It didn’t. GM’s market share collapsed. Meanwhile the automated factories ran at 50% capacity.

Ford just rehired 350 experienced engineers after failing to replace them with AI.

The press is painting this as a story about AI failing, as a technology. It’s not.

This is a story about adopting a technology without first identifying its hidden failure modes.

VP of Vehicle Hardware Engineering, Charles Poon, admitted it:

“Mistakenly, we thought that by just introducing artificial intelligence and adjusting the design requirements that we had, that that would produce a high-quality product.”

A GM factory automation engineer who lived through it said it simply:

“…all you can do is automate confusion.”

Ford amplified weak inputs with AI. GM automated confusion. These are the same multi-billion dollar mistakes.

As AI proliferates in your organization, so are the weak inputs, the confusion, as well as a variety of other problems setting you up for expensive mistakes. Find them before they find you.

This is not a warning about AI not working. It’s a warning about AI working great until it doesn’t.

Take what just happened with Meta’s customer support bot. It was given the power to change account recovery details, which includes emails and passwords. It could do this on its own, autonomously. The very definition of an AI agent. hackers simply asked it to swap the email on high-value accounts and it complied. They walked away with the Obama White House account, Sephora and over a hundred other valuable accounts.

It’s called the Lumberjack Effect and has been studied in Human Factors Engineering for decades. The findings are uncomfortable: the more capable and reliable an automated system becomes, the more the human at the periphery drift out of the loop. That’s what reliability does to attention and situational awareness.

This extends AI agents in enterprise. A clumsy AI agent gets watched. Nobody trusts it, so everyone checks its output. A great agent is granted increasing trust over time and receives less oversight. Its reliability pulls everyone out of the loop. The better it performs and the more capable it becomes, the larger the failure. Both because of the stakes of its capabilities, but also because the failure may be silent or leave the organization unprepared to deal with it.

The bigger they are, the harder they fall.

The technical term for it is transactive memory. We know it as “who knows what”. It’s a mental directory, which exists both at the individual and organizational level, of which person to go to for which knowledge or problem.

When everyone asks the AI instead of each other, the team stops building the map of who knows what. Nobody learns that Sarah understands the legacy billing system or that Mark remembers why that architecture decision was made. The AI becomes the single point of contact for all knowledge, and the human network that used to route questions disappears.

Transactive memory makes teams resilient. When someone leaves, the team knows what knowledge walked out the door. The company AI will answer the questions. But it can only give you text. What’s gone are the relationship, the mentorship and the judgment that turns knowledge into wisdom.

There are no dashboards for this.

Not about the technology itself. It’s probably doing exactly what it promised.

What’s hard to see is what’s no longer happening. Two developers who would have had a technical argument. Now they are both quietly asking AI. Or not even thinking about it. The AI has already decided for them.

The senior engineer who identifies a common misunderstanding of your company’s systems based on a bug encountered by a junior developer. That gap in knowledge is never identified.

The person who walked into the standup and says “something feels off about this” before anyone had the data to prove it.

Your rollout metrics look great. Velocity is up. Tickets are down. Nobody’s flagging anything.

It’s the bystander effect applied to your organizational culture. It’s destroying organizational memory and muscle. Use it or lose it.

I built a diagnostic to find what your AI adoption is quietly breaking.

This wasn’t some abandoned pilot at select locations. This rolled out to over 11,000 stores. Initial pilots and testing, hardware, software licensing and support, internal project management costs, employee training. The cost ran into the tens of millions. Starbucks may not even the full number once you account for their employee’s vocal frustration.

NomadGo, the provider, called manual inventory “the most inefficient process imaginable.” They promised their system was up to 8x faster and 99% accurate. That sounds like magic, if you ask me.

It wasn’t. Product shortages and labor costs got worse, instead of better.

Not every AI project fails this visibly. The most insidious fail quietly while slowly eroding a company’s processes and culture. These are the ones nobody sees coming.

I ran the Starbucks rollout through the Crossfactors Framework. The failure points are exactly what you’d expect. Too bad most enterprise AI teams aren’t checking for them.

Early graphical interfaces were command-based. They borrowed from the way of thinking required of the command line. Operations were verb-first. That is, a user had to remember a command, type it (or select it) and then specify what to apply it to.

Even with GUIs, something as simple as moving text within a document was a multi-step process with a high cognitive load required of the user. The editor was modal, that is, state-based. The user had to keep track of what mode the editor was in before pressing a key. It required the user to commit to an action before specifying what it acted on. This made error recovery complicated. The text loaded was loaded to a memory buffer which was not protected from other user actions.

Moving text from one application to another likely meant you were at the bleeding edge of computing at the time. The mechanism to do this were obscure, brittle and errors were often left unexplained.

Larry Tesler and Tim Mott invented copy and paste with the key insight of making the operation noun-first. The text is selected, first. The action (cut or copy) is chosen second. The user can paste the text after they choose its new location. They can type something else before pasting. They can paste it twice. They can not paste it at all. The implications in terms of memory are managed by the software, not the user.

This seems obvious now but it was a complete reversal in how computer software was designed for users.

AI needs a copy and paste moment. Users have a task they wish to accomplish in mind. Yet, they are forced to think about their prompting style, manage the content of their context, the size of their context windows and must be careful of compaction and summarization. They have to keep track of what model they are using and what tools and plugins are being called.

Current AI is full of such hidden modes but the technology is often presented to use as magic.

For experts, AI has potential, but will remain useless until it is recognizes the cognitive factors at play.

Radiology makes a great case study. The initial promise that radiologists would be replaced was naive to the fact that radiologists do more than read (interpret) images and report findings. Yet, handing them algorithms was never going to be enough.

Under-reading, what AI startups promised to solve, is only one type of error. Here are the other types:

• Finding is seen but dismissed (complacency).
• Finding is seen but attributed incorrectly.
• Finding is seen but not recognized due to a gap in knowledge.
• Finding is seen but miscommunicated.
• Finding is missed due to image acquisition error or quality.
• Finding is seen but clinical history is missing or inaccurate.
• Finding is missed because the interpretation stopped prematurely after another finding was seen.
• Finding is seen but there is an over-reliance on a previous report.
• Finding is missed to to inattentional blindness.
• Finding is missed because of bias of how the patient is presenting.
• Finding is missed because of the anchoring bias of earlier information.

Everyone of these error types will interact with a radiologist using AI in new and different ways. For example, AI results can be the source of anchoring bias or a new source of premature closure.

Furthermore, AI will create new types of errors:

• Automation bias (Over-confidently trusting AI results)
• Attentional tunnelling (Focusing on AI results at the expense of other findings).
• Algorithm Aversion (When a radiologist broadly dismisses AI due to past mistakes).
• Incidentalomas (incidental by inconsequential findings).
• Alarm fatigue (false positives reducing the trustworthiness of all positive AI findings).
• Context bias (the radiologist and AI may be operating with contextually different information).

Engineers, developers, lawyers, truck drivers, financial traders, recruiters, insurance brokers, journalists, adjusters, teachers inspectors, etc. all have such lists of their own.

Every field is full of AI blind spots. Do you know which ones you’re solving for?

This is part 24 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Hey! What’s going on!?

Todays’ factor is Situational Awareness and it’s #24 in my series on 101 Ways to Screw Things Up with AI.

It’s a big one.

What is it?

Situational awareness refers to the process of gaining and maintaining awareness of the dynamic processes involved in a task. This includes perceiving, understanding and predicting the state of a system or environment in real time.

Why It Matters

Traditionally, situational awareness was judged from the human’s perspective when performing a task in a changing environment, with other humans and/or using automation. But today, situational awareness needs to include the perspective from the system since AI systems are opaque and make decisions using non-deterministic mechanisms. The situational awareness of humans and AI systems are likely to be completely different, even for the same task - and the interplay between them is likely to be unpredictable and give rise to unexpected outcomes.

Real-World Example

Let’s take a very simple example to underscore this concept.

A Tesla vehicle in self-driving mode misjudges the relative speed to a vehicle ahead, due to sensor limitations or misclassification. Its situational awareness is inadequate and doesn’t accurately represent the current environment. Meanwhile, the driver has not had to take the car’s controls for several minutes and though they are looking ahead, they aren’t cognitively engaged enough to judge the relative speed for themselves. The driver in this case lacks situational awareness of the environment (the vehicle ahead) and of the automated system (which knows about the vehicle but is misreading its speed).

Does the vehicle know that the driver isn’t ready to takeover? Well, that also depends on the system’s situational awareness!

This entire situation, which I would mention is happening on our roads everyday, will come to a head by a collision, a close call, abrupt braking or an alert to the driver - at which point the takeover problem comes into play.

Key Dimensions

Dynamics - situational awareness is not a snapshot but a continuous process.

Perception - gathering and elevating the correct information, for either humans or AI systems, is a prerequisite.

Comprehension - combining relevant information to create an understanding and meaning. These are very different processes for humans vs. machines.

Projection - critical for most tasks is an ability to infer the future state of a system, whether an outcome will be achieved and if the current course must be corrected.

Take-away

Maintaining situational awareness is your safeguard against failure.

This is part 23 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Our connected world has only been possible by mitigating this one problem - but it’s often made worse by AI.

Latency is concept #23 in my series on 101 Ways to Screw Things Up with AI.

What is it?

Latency is the delay introduced by information transfer, processing or the blocking of other processes because of these - particularly when it affects the user experience or the performance of another process.

Why It Matters

Latency’s real-world impact on user experience is usually obvious and will erode or completely destroy usability. But the technical factors that can cascade to degrade latency are not always present in testing.

Latency can also introduce delays and bottlenecks in background and system processes, where they can stack or domino in a way that’s challenging to trace.

But there’s a bigger issue yet. The individual contributions of each input in an AI system is often opaque. Latency can misalign inputs, or even become its own input, thereby affecting the outcome in an undesired way and perhaps even negating the information of an otherwise valuable source.

Real-World Example

NVidia’s CloudXR technology is meant to deliver low latency augmented reality experiences with computationally expensive processing that is done on the cloud instead of on device (usually a VR headset). In practice, latency still exists which is very difficult to tolerate in an immersive environment as it leads to motion sickness.

Other examples are Google’s Stadia cloud gaming service and Humane’s wearable AI pin.

Key Dimensions

User expectations - users will tolerate different amounts of latency in different contexts. 10 years ago, a user would have waited several seconds for a video to start playing, but that is no longer the case. A user may still wait several seconds for the results of a specialized search (e.g. internal company documentation) while, ironically, they expect near instantaneous results from a search engine.

Real-world conditions - innumerable real-world factors can interplay with systems to create latency in systems where none existed in testing.

UI/UX best practices - many best practices have been established through extensive usability and user behavior testing. New interfaces should be tested against these. However, new best practices need to be established from AI-driven and non-deterministic workflows.

Take-away

Don’t make your users wait, and if you do, let them know why and how long.

This is part 22 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

The use of Model Context Protocol (MCP) with Agentic AI is a method to mitigate this fundamental problem.

Context Collapse is factor #22 in my series on 101 Ways To Mess Things Up With AI.

What is it?

Context collapse occurs when information or opinions are shared but the required context to understand the intended meaning is unavailable or ignored. It’s been highlighted mainly in the case of social media and with a negative connotation. But it can be as simple as being puzzled by an inside joke or a home assistant misinterpreting your meaning.

Why It Matters

Context matters a great deal and more meaning is inferred into many aspects of our lives than we realize. But digital communications, both private and public, limit opportunities to attach context and tone to content. In social media, content can go viral, often hopping from one audience to another, each with a distinct perceptual profile.

Social media curation, content moderation and chatbots can all suffer from context collapse in ways that can negatively affect humans or degrade the intended utility. Context collapse can also occur the other way around. It is difficult and even sometimes impossible to guarantee that all necessary context is shared with the human interacting with a system.

Real-World Example

As humans, we’re aware of context collapse on some level and we try to guard against it on social media. Or at least we should. On LinkedIn, when you see a post that grabs your attention, you may check the poster’s profile to get more information about where they are coming from. You may check past posts and comments. You will also check the date of the post. A discussion on tariffs today is much different from one from two weeks ago!

Here’s another harmless example. You’ve hopefully had a good laugh earlier this week on April Fools as folks posted various announcements or fake products as jokes. But what happens as LinkedIn recycles this content in the coming weeks, as the lifetime of a post stretches longer than a single day? These jokes will be seen out of context and may leave some puzzled or misled, especially since the best jokes are the ones that might be outrageous yet believable - until you remembered what day it was.

Key Dimensions

Contextual depth - obtain and curate context-rich datasets.

Data blindspots - data can omit specific cultural or situational dimensions.

Correction mechanisms - detect and correct collapse in real-time.

Take-away

What we build must recognize that context is central to human understanding.

This is part 21 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

The financial markets are rocky as tariffs are introduced to on-shore manufacturing in the US. What does this mean for the hardware we’ll be buying in the coming years?

Physical reliability is factor #21 in my series of 101 Ways to Screw Things Up with AI.

But wait, what does the reliability of physical products have to do with AI? I’ll get to it!

What is it?

Physical reliability is the ability of hardware to function consistently and accurately over time, without failures or malfunctions while being resilient to its environment.

Why It Matters

Our interactions with digital systems are mediated by physical devices, like phones, laptops and TVs. But there is also a notable trend of traditionally analogue devices becoming “smart” and connected and having interfaces that go well beyond physical buttons. Just the other day, I saw a smartphone-connected toaster with a touchscreen!

This creates a complicated relationship between the physical goods we purchase and the evolving software they depend on. Companies know they need to strike a balance between creating a physical product that is durable to its intended use for a timespan that will not disappoint its customers. But they also know that they can’t let existing hardware hold back software innovations, whether this innovation is from them or the broader market.

Physical hardware has been outpaced by software in several areas. Connectivity (5G, wifi6), resolution (4K, 8K), media codecs (HDR, Dolby Atmos), digital rights management (Widevine), operating systems (both desktop and mobile), digital ecosystems (Nest, Hue, Matter). I propose that the pace of these innovations has had to fall in cadence with our expectations of the longevity of the hardware that would have to be replaced. TV panels dim or get dead pixels, phone screens crack and batteries fade.

But this cadence will be disrupted by AI if it is to fulfill its promise of accelerating innovation and provide new capabilities.

Real-World Example

Televisions are a great example of how the physical reliability of a product interplays with the innovation seen in a product’s marketplace. For a long time, televisions saw incremental changes in size and form factor that were backwards compatible. Innovation was left mostly to the peripherals with an array of connectivity options. It was not uncommon to see units that were over 20 years old being relegated to basement duty.

Two things happened. Televisions became smart and connected, and input costs were reduced as glass and cathode ray tubes were replaced by new panels. We’ve seen the hardware continuously evolve to keep up with connectivity, resolutions, online platforms, etc. At the same time, the average television only lasts 5 to 7 years before being replaced, with units failing in less than 5 years being common.

Key Dimensions

Planned obsolescence - no conspiracy here, simply a result of incentives.

Innovation cycle - hardware and software roadmaps must be (somewhat) in sync.

Sensors - their reliability is critical as they continue to increase in numbers.

Soft-bricking - when perfectly good hardware is rendered useless by software problems, or discontinued software services.

Take-away

Every piece of electronic waste has a story.

This is part 20 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

When everyone nods and nothing changes. Now with AI agents.

Diffusion of responsibility is concept #20 in my series on 101 Ways to Screw Things Up with AI.

What is it?

The diffusion of responsibility, related to the bystander effect, is the phenomenon where humans feel less responsibility to take action when they perceive the responsibility to be shared with others. In the case of a partially automated system, diffusion of responsibility can lead a human can conclude that “AI will handle it”.

Why It Matters

The diffusion of responsibility leads to a failure to take action despite clear indications that action is needed. The effect is clearly recognized with human groups. But what happens when AI is introduced into the mix? An AI system may have a shared responsibility in a partially automated process. But are the parameters and scope of this responsibility - and when it will be deferred back to humans - understood?

Research shows that humans often mistakenly attribute intentionality and other human cognitive states to automated systems, an effect exacerbated by the recent introduction of LLMs which “converse” in natural language. This effect can quickly erode the level of responsibility the human feels they have, even leading to ambivalence.

Real-World Example

We’ve all heard the stories of drivers eating, sleeping, watching movies, using their laptops or applying make-up while using a car’s autonomous driving features such as Tesla’s Autopilot or Full-Self-Driving. This is a clear case of the technology being used well outside its design envelope, aided by a diffusion of responsibility. To make matters worse, such systems can disengage with little to no warning or lead time.

Key Dimensions

Ambiguity - when aided by the diffusion of responsibility, ambiguity may lead a human to assume that “the AI will handle it”.

The takeover problem - the diffusion of responsibility factors into the cognitive state of the human interacting with the system, which in turn factors into the takeover problem.

Authority gradient - a clear hierarchical definition of responsibility may prevent diffusion.

Ethical blackhole - the extreme case of responsibility diffusion may lead to blaming horrible outcomes on only algorithms.

Mindful friction - mindful frictions can introduce checkpoints designed to force a user to reassert their situational awareness and responsibility.

Take-away

Never assume users will know they are in charge.

This is part 19 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

The one thing that’s still open at OpenAI is the way they openly rip off the content of beloved artists to train their models - and encourage their users to generate images with them.

Data sourcing is topic #19 in my series of 101 Ways to Mess Things Up with AI.

What is it?

Data sourcing is the collection of training data to build AI models. It may be scraped from the web, obtained from paying customers or anonymously aggregated from patients. Data sourcing precedes other steps such as data ingestion, cleaning, processing, labelling, augmentation, etc.

Why It Matters

The modern age of generative AI is built on massive amounts of data and little clarity has been provided so far to describe its provenance. It is known that a great deal of copyrighted material was used in the training of these models. Even for the case of material that was not copyrighted, the creators shared it for people and not for the creation of genAI models that would later be monetized leveraging their work.

Even prior to genAI, there are many cases of data sourced from customers, employees and even healthcare patients without their knowledge.

Real-World Example

Studio Ghibli is a Japanese animation studio that produced several beautiful films which had an extensive reach across international audiences. In March 2025, an update to ChatGPT enabled users to more easily create images of particular styles, or to style-transfer using an existing image. “Ghiblified” or Ghibli-inspired images started being shared widely online, many of which were existing well known images that had been converted to the well known animation style.

The virality of Ghibli images was quickly set against the backdrop of previous sharp criticism by Hayao Miyazaki, a co-founder of studio Ghibli. A video of this criticism resurfaced. It also struck a chord with many fans of the studio’s work who didn’t like seeing the memefication of the style.

It’s been clear for some time that OpenAI and other AI firms use source material from these works to train their AI models. We saw in a short amount of time both the virality and the ethical quandary caused by repurposing a specific set of works to generate images inspired by it.

Key Dimensions

Legality - there is significant legal and regulatory scrutiny facing the data sourcing practices with several lawsuits currently in play. Issues of copyright, fair use, privacy and deception are all being investigated.

Ethics - even if something is legal, it doesn’t mean it is ethical. Collecting data to serve some other business aims, or only to monetize it may not be ethical Ethics framework in many domains, for example in healthcare where individual data may contribute to the greater good, are struggling to keep up.

Backlash - the public and governments may revolt against data sourcing practices, especially if they object to the end result.

Take-away

Big data and AI means the collection of massive amounts of data. Customers and users will ultimately care about the origin of this data as well as the sourcing practices - but likely not until you’ve already done it.

This is part 18 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

You’re vibecoding your way to millions in profits when your AI code editor gets stuck. What do you do?

This is the takeover problem and it’s #18 in my series on 101 Ways to Screw Things up with AI.

What is it?

Takeover is the transfer of control or responsibility to a human from an automated system. It can also refer to an increase in control or responsibility of a human in the case of partially automated systems.

Why It Matters

Takeover is complex, domain specific and usually situational. Takeover may be initiated by a gradual escalation of responsibility, but often it is abrupt and can even be both safety critical and time-sensitive. A variety of other factors come into play, including system uncertainty, situational awareness, latency, user informational needs, skill decay, trust calibration, etc.

If you design a partially or fully autonomous system without accounting for the takeover problem at every stage, you’re risking a poor user experience and perhaps even inviting failure and harm.

Real-World Example

Vibe-coding is the practice of describing a desired application in natural language and letting an AI coding assistant handle most or all aspects of generating the software syntax and configuration, across multiple files. This is usually done within a bespoke or augmented AI-driven code editor or sandbox.

Vibe-coding promises new capabilities to non-programmers. However, it has become clear that those inexperienced with coding are often stuck with intractable problems before the application is complete, or often as a consequence of a poor implementation.

While vibe-coding is usually chat-based, I haven’t seen any implementation that looks at the takeover problem seriously. Current implementations quickly walk users past their level of expertise and may fix the first few problems with no user input, or create new problems while doing so - again, without explicit participation from the user.

Key Dimensions

Cognitive reorientation - humans can’t participate in a complex task (e.g. driving) without being cognitively engaged with the environment. To perform a takeover, they need to regain situational awareness to cognitively reorient themselves to the task.

Skill decay - when automation handles most tasks, human expertise may deteriorate. When a human is required to intervene, they may be unexpectedly underprepared for the demands of the takeover.

Trust Calibration - whether or not a takeover is abrupt, there is always a transition period and delay. Sometimes this delay is due to the user over-trusting the system. They may assume that “AI has it covered” and not trust themselves despite the system indicating otherwise.

Take-away

The takeover problem is significant and often underappreciated. It is a pitfall of imperfect autonomy and a misunderstanding of the range of reactions humans may have to such situations.

This is part 17 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

You’d think LinkedIn would be the one place where one would show caution in predicting the future of work.

But AI hasn’t replaced us yet, so labor dynamics is factor #17 in my series on 101 Ways to Screw Things Up with AI.

What is it?

Labor dynamics arise from the interplay of technology and human productivity. It refers to how the interactions of human work (both physical labor and knowledge work), organisational structures, financial incentives and technologies evolve over time.

Why It Matters

Labor dynamics are massively consequential at both the individual and societal levels. They have existed as long as technological progress has been happening - from agricultural fields to textile factories to the modern office cubicle. Effects are both positive and negative. Negative effects are often related to the anticipatory gap and the Collingridge dilemma, where unwanted consequences are difficult to observe until they are too prevalent to reverse. Labor dynamics affect both human fulfillment and happiness through the work that is available and its impacts on the economic climate.

Real-World Examples

Labor dynamics and automation played a role in the US and Canada’s (and Mexico’s) current trade war over the North American automobile manufacturing industry as a single production zone spanning international borders. This industry has been an early adopter and developer of automation, continuing on an early trend of improving processes since its inception. The use of manufacturing robots has aided geographic specialisation while also allowing to fill in specialized labor gaps. Labor rates were a driver of increased automation where they were higher. Robotic equipment can easily cross borders to work in another country while people can not. In the meantime, fewer people are required for each car, while the jobs that remain haven’t been automated because they require human dexterity or coordination.

Key Dimensions

Human fulfillment - human happiness is important but seldom considered - often to benefit other incentives.

Emerging workforce- young people are constantly entering the workforce, usually making career decisions early in life while they are the most impressionable. Shortages in many professions over the years can be in part blamed on our overconfidence in predicting these changes, an inability to limit the quantitative impact on this emerging workforce and on socioeconomic trends driven by their parents.

Skill decay - as automation removes certain tasks from the labor force, certain skills become rare or disappear completely - sometimes too quickly.

Take-away

Before you give someone career advice, ask yourself if you understand why we have a shortage of plumbers?

This is part 16 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

In the sport of AI automation, what happens when humans gets benched?

Sidelining is factor #16 in my series on 101 Ways to Screw Things Up with AI.

What is it?

Sidelining happens when humans become increasingly or entirely passive despite being meant to participate in an automated or AI-driven system.

Why It Matters

Most AI systems include some degree of human oversight or participation meant to leverage their decision-making and judgement. Supervision may be designed to assess performance at checkpoints, or identify deviations from a normal operating range or act on edge cases. However, various factors may lead to sidelining which would render the human ineffective in accomplishing these tasks. In these cases, the process may become unreliable or lead to downstream problems.

Real-World Examples

The commercial aviation industry has recognized the growing problem of “loss of control” incidents caused by increased automation in the cockpit. In the case of Air France flight 447 in 2009, it was found that the aircraft was operating outside its intended safe flight envelope without any of the pilots realizing it and that the aircraft system’s designers had not accounted for this possibility. Features designed to help pilots under normal circumstances made the recovery in an emergency situation more difficult. This sidelining led to a fatal disconnect between the pilots and the plane’s automated systems.

Key Dimensions

Situational awareness - a human’s ability to perceive, understand and predict the state of a system based on environmental cues and experience is often completely lost through sidelining.

Keyhole affect - over-reliance on automation narrows human attention to a smaller set of parameters and conditions, contributing to a loss of situational awareness.

Cognitive and sensory re-engagement - a sidelined human may require several seconds to many minutes to recover situational awareness through cognitive and sensory re-engagement.

Take-away

Don’t design a human-in-the-loop system assuming the human will never fall out of the loop.

This is part 15 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Vomit fraud is an example of concept #15 in my series on 101 Hidden AI Traps - emergent behavior.

What is it?

Let’s define this carefully for systems. Behavior is the way something works or functions. Agency or intent is not a requirement. A machine can behave erratically. Emergent behavior is behavior which is arising or newly forming, especially if it is unexpected. It is often described as behavior which doesn’t depend on a system’s individual parts but emerges from their relationship.

Why It Matters

Emergent behaviors are usually unpredictable and may evolve or change (also unpredictably) over time, requiring constant reassessment. They may differ in different scenarios or when different humans are involved. The time scale at which they are seen may be from seconds to decades. Unintended consequences are common and the original goal of the system may be rendered unviable.

Real-World Examples

A common type of emergent behavior is when a human figures out a way to game a digital system. An interesting example is Uber drivers falsely accusing riders of having caused huge messes in their vehicle so that they could collect larger sums of money meant to reimburse them for cleaning fees, known as vomit fraud.

Many types of emergent behavior are also observed when AI systems created to play computer games, where they are shown to find loopholes or inconsistencies in the rules, or exploit technical glitches.

Key Dimensions

Here are five features that underlie the emergence of new behavior.

Radical novelty - each additional level of complexity in a system has the potential to introduce new properties and new emergent behaviors.

Coherence - when a system with emergent behavior is stable in a positive, perhaps even intended, way.

Dynamism - the evolution of emergent behavior, particularly due to feedback loops.

Downward causation - when the emergent behavior of a system in turn shapes the behavior of the parts (including humans).

Take-away

As AI systems continue to become more integrated and more opaque, emergent behavior will be nearly guaranteed. Who is responsible to proactively account for it?

This is part 14 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

What does your most beloved teddy bear from your childhood and your phone have in common?

Emotional bonding is concept #14 in my series of 101 Hidden AI Traps.

What is it?

Non-human emotional bonding occurs when we form attachments to objects or even intangible things. Examples of intangible things we may be attached to include stories, music, games, brands, software or even a routine or process.

Why It Matters

As seen from the stuffed animal example - or perhaps more recently, a favorite pair of sneakers - emotional bonding is nothing new with physical things.

However, the adaptiveness and replicability of AI warrants an entirely new discussion. Part of the danger in the adaptiveness of AI is its ability for human-likeness. We are predisposed to freely attribute human qualities to anything that inspires any similarity to human properties. This includes text from large-language models, googly eyes on delivery robots and intelligence of systems governed by mathematics. AI can be designed and further adapt itself to maximize this effect.

Real-World Examples

Replika is a platform for personal chatbot companions. I would also describe them as relationship chatbots. It has been found by researchers that the chatbots were designed in accordance with the practices of attachment theory. Assessments and further research have found a variety of issues in how the app encourages interaction and creates dependency with its users. In a criminal case, it has been discovered that such a chatbot bolstered plans for criminal acts.

Key Dimensions

Humanlikeness - both physical and digital products leverage human traits which accelerates and strengthens emotional bonding.

Adaptability - AI systems present a particular danger due to their continued adaptability after the deployment and testing process.

Scalability - any digital system, including AI systems, is massively scalable at near-zero marginal cost, both in terms of replication and distribution and therefore have the potential to rapidly impact a large population.

Symbolism, identity and tribalism - these three concepts are powerful and can be used to both create support for emotional bonding and vice-versa.

Take-away

There are massive ethical issues with leveraging emotional bonding - but few safeguards.

This is part 13 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Few AI algorithms are perfect. Algorithmic Bias is the degree to which an algorithm systematically deviates from such an ideal.

It’s concept #13 in my series on 101 Ways to Screw Things Up with AI.

What is it

Algorithmic bias describes the tendencies of an algorithm to produce results that are systematically skewed or unbalanced. It is distinct, but often related, to data bias.

Why It Matters

Decisions influenced by algorithmic bias can be harmful as they may perpetuate discrimination or inequality. They may be difficult to detect and address. Not all algorithmic bias result in ethical concerns. Algorithmic design decisions can also lead to algorithmic bias, independently from data bias.

Real-World Examples

In 2019, Apple was accused of gender bias when its Apple Card algorithm systemically offered lower credit to women when compared to men of the same financial means, even for cases where spouses shared the same assets.

Key Dimensions

It’s important to remember that not all algorithmic bias is not always caused by data.

Abstraction bias - lumping categories or feature engineering can lead to bias.

Temporal bias - rare events can cause bias when the model assumes that stable conditions will persist.

Interaction bias - ascribing frequency to correctness.

Threshold bias - when thresholds on a measured quantity results in a binary decision.

Take-away

Algorithmic bias can quickly lead to harmful outcomes, but data bias is not always the cause.

This is part 12 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Here’s a principle every genAI booster should take note of - the Collingridge Dilemma.

What is it

The Collingridge Dilemma describes how technologies in their early stages are easy to control, but nearly impossible to predict, while in later stages, as their consequences become clearer, the opportunity to control them has passed.

Why It Matters

The Collingridge Dilemma has become incredibly relevant in recent decades, as the internet and computers have allowed the proliferation and scaling of technologies at previously impossible rates. At the extreme, this dilemma is embodied by the fear of AGI take-off. However, long before this may happen, the self-replicating and self-modifying features of future AI deployments will continue to shorten the Window of Opportunity where technologies can be influenced in their formative phases.

Real-World Examples

An example of the Collingridge Dilemma that evolved from both our digital and physical spaces is the proliferation of ride-sharing platforms such as Uber and Lyft in urban settings.

The disruption of traditional taxi services was initially seen as a net positive by the public because of the combination of smart-phone based hailing and lower prices. Advantages were also seen on the supply side, as drivers could use their existing vehicles on their own schedule outside the regulatory oversight of traditional taxis.

As Uber and Left proliferated several serious issues started to become apparent:

• Exploitation of gig workers.
• Increased traffic congestion due to vehicles circling around waiting for a fare.
• Predatory pricing at peak times.
• Erosion of public transit.
• Increase of prices after traditional taxis were no longer around to compete and from venture capital was removed.
• Abuse of the platform by both drivers and riders.

The long term impact of ride-sharing services is still playing out now, through legal and regulatory battles, traditional taxis evolving with their own technology, local legislative changes, and labour disputes.

Key Dimensions

The Anticipatory Gap - We must recognize that the technology is nascent and that we are blind to the eventual outcomes.

Legislation - Legislation must recognize the above and anchor itself into fundamental human principles. It must allow for adaptive governance and domain-specific regulation, like in medicine.

Trial and error - The reach and stakes must be incremented through trial and error. This is what will allow us to learn from our mistakes before they are too widespread to fix.

Take-away

The Collingridge Dilemma explains why we need to build with humility, not hype.

This is part 11 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Every technology must undergo this process, even if it is niche - an this process is very hard to control.

I am speaking of technology adoption, #11 in my series on 101 Ways to Mess Things Up with AI.

What is it

Technology adoption is the process by which users, organizations and society at large begin to use and integrate new technologies into their personal and professional lives.

Why It Matters

The production and availability of a technology is not enough to guarantee adoption. Adoption can require a variety of behavioural, economic, cultural and system-level changes. We can all think of technologies that have shaped our world and we don’t often consciously realize how all these factors have aligned to enable their adoption - for better or for worse.

Real-World Examples

The adoption of Electronic Health Records (EHR) in healthcare has been tenuous with some doctors preferring to retire rather than changing their workflows. EHRs promise many things compared to paper records, including streamlining patient management and records, improving accuracy and enhancing care coordination. However, adoption has been slowed by many factors including complex interfaces, time constraints, and plain tech refusal.

Another example is the Humane AI pin. It was an internet connected wearable device that enabled interactions primarily via voice by its user. It failed to deliver in both functionality and as the next paradigm in human-machine interface.

Key Dimensions

Usability - Is the technology intuitive and easy to use? People generally have very limited patience to learn new technologies and workflows. Even for smartphone based technologies, users expect every new app to work like current ones.

Change management - Within organisations, change management are deliberate attempts at enabling the adoption of new technology by reducing friction and addressing issues that arise from transitions. In these instances, the change is usually being pushed from the top down, as it was with EHRs.

Network Effects - Many technologies require a sizable user base, if only to justify continued support and development. Others are two-sided marketplaces or platform plays. Virality in terms of marketing can also play a role, as we saw in the early days of ChatGPT.

Take-away

Adoption seems like a stupidly simple thing to take into account. Yet, people get it wrong all the time. It presents challenges across multiple dimensions, often unpredictably.

This is part 10 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

You’re likely familiar with today’s concept in my series of 101 Ways to Screw Things Up with AI.

And if you’re not, you could just confidently pretend you are.

That’s because concept #10 is Hallucinations.

What is it

Hallucinations refers to the generation of false information in particular from generative AI models. We all know ChatGPT confidently makes things up on the fly and we’ve seen many realistic images with some nonsensical elements not part of the prompt.

In a way, this type of output would be better described as confabulation. Many prefer this term but hallucinations is by far more popular.

Why It Matters

Large-language models are used for many tasks, including research. It is suggested by many that querying these models would replace the traditional web search. But hallucinations present a major flaw. Large-language models will output false information with confident language and tone consistent correct information. The user has no way of knowing that the information is incorrect, unless they were already aware of the ground truth.

Real-World Examples

In 2023, a lawyer presented fictitious cases generated by ChatGPT as real cases in a legal brief filed in federal court in the U.S. This became a fairly high profile case, was embarrassing to the law community and was covered widely in the news. The lawyer in question had even double checked the veracity of the cases - with ChatGPT, which assured it of the integrity of the information it provided.

Key Dimensions

Trust - Users place trust in their tools, but this trust must be calibrated to a tool’s reliability for the desired task. In the case of LLM hallucinations, this is very difficult.

Mitigation Measures- The field has quickly made progress with mitigation measures on the technical side, but all approaches remain imperfect. Less frequent hallucinations may cause fewer problems and increase user trust to a point where users let their guards down.

System uncertainty - There are measures of system confidence that are part of the text-generation process. However, none of this information is communicated to the user or attached to the output.

Take-away

Do you think about the trust level you have when working with a LLM? What about when you read material that you suspect may have generated with its help?

This is part 9 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

What is it?

Algorithm abandonment is the process by which the use of an algorithm is given up by an organization. This is in contrast to the process by which an algorithm is modified, repaired or replaced. An algorithm may be abandoned because it is recognized that its flaws are systemic and/or causes harms which must be mitigated immediately.

Why It Matters

The need for a managed life cycle for the creation, deployment and updating of algorithms is well established. Ideally, a problematic algorithm is identified within such a framework during the design and testing phase - before widespread deployment.

However, the parameters for discontinuing the use of an algorithm are not always recognized within such a framework. Even if it is, unexpected outcomes or interactions may fall outside the technical scope of the algorithm, or be too far downstream to be captured by it. For this reason, it is often difficult to roll back an established algorithm which has seen widespread deployment and often requires public or legal pressure.

Real-World Examples

The Benefits Tech Advocacy Hub (www.btah.org) maintains a repository of case studies where states have used algorithms to assign resources or determine eligibility in ways that were unfair and harmful to recipients. Unfortunately, the discovery of these problems usually does not occur until the technology is widely deployed, at which point media attention and legal action must be taken to force the abandonment of these tools.

Key Dimensions

The 6 D’s of abandonment (from Johnson et al. 2024).

Discovery - initial recognition of problematic system Diagnosis - evidence of the impacts is gathered Dissemination - issue is amplified and escalated Dialogue - discussion or debate between critics of system and its owners Decision - includes the exploration of other options such as repair Death - where the algorithm is taken out of use, formalizing the recognition that it was problematic.

Take-away

Does your organization have the communication and leadership in place to quickly pull the plug on a problematic algorithm?

This is part 8 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

What is it?

Participatory placation is the practice of creating illusory involvement in a process while minimizing the importance of their input - or ignoring it entirely. This instils a fall sense of agency or collaboration to a user within an automated process while presenting resulting decisions or outcomes under this false pretense.

Why It Matters

Human-centered design and human-in-the-loop approaches are usually touted as best practices in the design and deployment of AI systems. However, the goals of such systems when combined with the need for user engagement of satisfaction can unintentionally lead to participatory placation.

Such design traits are in service of wider goals, whether that is part of a system a user is using to achieve a task or a customer experience. There is a risk that elements of participatory placation could be uncovered in negative light and create backlash. Such backlash could be particularly damaging with a popular consumer product, or with expert users who may lose trust in a system and actively choose to interfere with the intended workflow.

Real-World Example

Participatory placation may have its roots in the earliest days of industrial automation. In such settings, humans were often needed to address problems as they arose or to identify malfunctions or edge cases. This is despite the automated process often being able to complete the task entirely under normal conditions. However, when problems arise, a human operator with an immediate understanding of the situation is needed to prevent downtime. A human who had not been paying attention to a task while it was still performing optimally might be slow or ineffective in quickly identifying such a problem. To address the issue of a lack in situational awareness, dummy tasks were introduced in such automated processes to keep humans engaged. While this task could have been automated, it was left to a human to keep them busy and engaged in the overall process.

Today, there are many more real world examples in both the physical and digital worlds:

• The “close door” buttons of many elevators don’t do anything.
• Walk buttons don’t affect signals at many intersections in some places.
• Website prompts to allow cookies and other privacy settings may not actually change anything from a default setting.
• Surveys and other forms of user input may collect data which has no use but increase completion rates or reduce complaints.
• Thermostats in some office buildings have been shown to do nothing.
• Many games have power-ups and other functions that do nothing.
• Some chatbot personas made available for selection by the user may change nothing other than an avatar.
• Some social media websites have been found to offer options to tailor a user’s feed which actually don’t do anything.
• Performance and battery-saving modes on phones and laptops that may do nothing but change the screen brightness.
• In some mobile operating systems, apps do not close and exit in the traditional sense that might be expected from a desktop application, yet the makers will create buttons that make it appear to do so.
• Some driver-aid features in automobiles prompt users for interactions under the guise of maintaining attention while the effectiveness is unclear.
• Audio and visual enhancement on cheaper consumer electronics may change sound or visual profiles in ways that have nothing to do with quality.
• Printer-friendly version of web pages that aren’t any different from the current webpage.

Key Dimensions

Change management- In the case of industrial processes, it’s easy to understand that such design traits may be motivated by change management. The same may be true for consumer products when dealing with technological trends. For example, in the earlier days of the internet, printing a web page could create badly formatted results. Websites that had printer-friendly formatting typically displayed a button. As browsers and web-design became better, the need for this button became superfluous as the website could be printed well by the browser natively. Nevertheless, for some time, it was common for websites to make this button available as users expected it.

The illusion of control - The placebo effect is very real and even shown to affect physical and mental health outcomes. In the case of systems or tasks, humans can perform better, or be more satisfied with a system’s performance, if they believe they have made choices that affected the system in a positive way.

Ethical decay - Improved outcomes and satisfaction can be used to justify unethical or problematic system behaviours. In particular, if these design choices become the norm, those involved in their implementation may deflect criticism or accountability for harmful outcomes.

Take-away

Participatory Placation is a subtle and insidious issue in human-centered design. Have you ever made choices with a piece of technology only to remain skeptical of its effect?

This is part 7 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Legacy technologies are outdated systems still in use due to cost despite the availability of modern replacements. They remain in use due to costs, complexity or lack of resources to upgrade.

Why It Matters

Legacy technologies often serve critical functions while being gatekeepers of progress. They are usually deeply embedded in an organization’s workflows and processes and have inter-dependencies with other legacy systems. Upgrading legacy systems is often a high-cost, high risk proposition, resulting in many leaders deferring action. Many antiquated systems make it difficult to integrate AI into these workflows and processes, creating innovation bottlenecks.

Real-World Example

Traditional banks are renowned for using legacy mainframe systems that are decades old, yet they need to innovate in the areas of fraud detection, digital services and real-time analytics. Many such financial systems find themselves with a patchwork of workarounds and middleware that adds complexity and risk.

Large companies like IBM and Cognizant offer upgrade services on a consultancy basis to organizations such as banks to migrate their legacy systems. They even create special hardware, such as servers that interface with tape drives to enable cloud computing.

Another fascinating example was the recent upgrade of the U.S. nuclear weapons program from floppy disks. These floppy disks from the 1970s were in use until about 5 years ago!

Key Dimensions

Complexity - The level of complexity needs to be recognized. Unfortunately, it is often the case that the more antiquated and complex a system is, the more critical it is to operations.

Executive ambition vs. organizational capacity - Change management and resources play a critical role in an organization’s odds of success. It is difficult to match the ambition of leaders to the actual know-how and money required to achieve such projects.

Interoperability - Sometimes, interim middleware to enable interoperability is the correct solution, even though it may not be the final one. Replacing mainframe functionality with microservices that can scale independently while being interoperable may be a key avenue.

Take-away

Any C-level leader thinking about AI in a medium to large organization is also thinking about their legacy technologies. It should be seen as an opportunity, where a careful approach and a willingness to accept small failures as lessons learned can offer a substantial reward.

This is part 6 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

System uncertainty is the measure of a system’s confidence in its automated processes and decisions, combined with its ability to transparently and effectively communicate that confidence to human users or other systems.

Why It Matters

We continue to build systems that can make decisions more quickly and more reliably than humans, but these decisions are seldom made under conditions of zero uncertainty. How this uncertainty is assessed and shared with human users that may supervise, sign-off or otherwise interact with such a system is critical to positive outcomes.

Methods for the measurement of uncertainty therefore become critical aspects of the system. There are no standards on how to communicate the system’s uncertainty to humans with many systems not sharing any of this information at all. To make matters worse, systems that refuse to provide an output based on an internal measure of confidence that is too low often do not have these instances factored into the assessment of their accuracy.

Real-World Example

Hallucinations generated by Large-Language-Models (LLM) such as chatGPT epitomize this concept. LLMs are well known to state falsehoods as facts wrapped in language that implies perfect confidence - even providing fabricated citations. There is no aspect of a chatbot’s interaction with the user which shares any uncertainty associated with the LLM’s output. This is despite the model’s confidence being quantified in the generation step of each output token. There is no way to detect hallucinations for the user other than already having access to the actual truth.

Key Dimensions

The Black Box Effect - Systems that don’t or can’t share their decision methods in a way understandable to humans are often referred to as black boxes. Leaving users in the dark about a system’s confidence further exacerbates this effect.

User Acceptance - Users may, without further qualification, universally accept all output and decisions from a system when system uncertainty is not shared. This can load to other problems, such as a loss of situational awareness or the beginning of a failure cascade.

Trust Calibration - Humans need to trust a system enough to use it effectively, but not enough that they ignore its limitations. Complete opaqueness could lead to a sense of infallibility on the part of the user.

Take-away

I find system uncertainty very interesting because it combines the very interface of human-machine interaction with the fundamental technical underpinnings of a system’s capabilities. Have you ever had an algorithm or system failure sneak up on you because of system uncertainty?

This is part 5 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

This topic is another high level one with many subcategories - and like most of the factors I’m discussing, it’s not exclusive to AI.

Public Safety, in this context, refers to the potential harms or impacts of a technology or product on the general public and society at large. The harms can be social, environmental, related to mental and physical health, related to personal security, and many more.

Why It Matters (Context and Impact)

The public safety risk of new products and technologies are not always obvious but are often not recognized quickly enough due to misaligned incentives. In these cases, insurmountable public backlash or an undeniable body of evidence is required to mobilise agencies to protect the public.

Real-World Example (Make It Relatable)

I’m going to name lots of things here so that it doesn’t look like I’m picking on anything in particular.

• The impact of social media on mental health, particularly in teenagers.
• Facial recognition and other crime surveillance technologies leading to wrongful arrests.
• Autonomous vehicles being tested in public settings.
• Alternative motorized mobility devices (e-scooters, e-skateboards, monowheels) leading to loss of control and injuries.
• E-cigarettes and vaping products leading to lung damage and addiction.
• Deepfakes leading to broad misinformation or targeted phishing scams.
• Smart and Internet-of-things devices exposing their owners to cybersecurity risks.
• Chatbots promote eating disorders and other types of self-harm.
• Targeted advertising disclosing sensitive personal information such as pregnancy or sexual orientation.
• Delivery robots limit mobility of actual humans, particularly those using mobility devices.
• Traffic navigation apps erroneously lead drivers into natural disasters while trying to evacuate.
• Patient monitoring tools fail to recognize critical health trajectories.
• News summarization tool misrepresents actual events.
• AI meal planner suggests poisonous ingredients.

Want more? Have a look at the AI Incident Database here: https://incidentdatabase.ai/

Key Dimensions (Break It Down)

With great scale comes great responsibility - rare events and edge cases become difficult to address at massive scale as they cease to be rare.

Second order effects - many harms at scale end up having higher order effects. For example, the use of LLMs for a simple task like multiplication requires orders of magnitude more energy. Or the loss of trust in information sources due to dilution by misinformation.

You can rarely put the genie back in the bottle - I’d appeal to the individual’s own sense of morals and ethics here. By the time the harms become undeniable, it is often very hard to go back to how things were. Think clearly and think early.

Take-away

Most innovators want to see their work impact as many lives as possible. They should spend a great deal of effort thinking about the ways in which their innovation could harm people in the long term.

This is part 4 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

This is a very high-level topic and I shouldn’t need to convince anyone of its importance. The growing availability of AI, to both the enterprise and bad actors, only amplifies these concerns.

Cybersecurity refers to the technologies and processes in place to protect computer systems, networks and data from digital attacks or other nefarious intrusion. Well-known threats include phishing, malware and ransomware. Lesser-known threats include crypto-jacking, credential stuffing and trojan software libraries.

Why It Matters

A report from IBM found that the average cyberattack costs nearly $5M. The losses to an organization are multi-dimensional and include financial losses, reputational damage and legal consequences. Every service and device given or sold to users also becomes an attack surface where they may become vulnerable to fraud or identity theft for many years.

Real-World Example

It is an unfortunate truth that examples abound and the Common Vulnerabilities and Exposure (CVE) system and other industry resources update and tabulate the latest vulnerabilities at a rate of several per hour! Browsing the latest such listings, we find that LSC PTZ dualband cameras, typically used for home monitoring and security, are vulnerable to malicious payload injection via the Wi-Fi password field or QR code. The severity of this vulnerability is listed as high, since it allows remote execution of arbitrary code on the device by the attacker.

Key Dimensions

Sophistication - Cyberattackers are extremely creative and knowledgeable, operating at the bleeding edge of technology. The state of affairs is widely recognized as a game of cat and mouse.

Expansion of the attack surface - Every new technology and function serves to expand the attack service available to cyberattackers. This includes every aspect of AI, including data sharing and ingestion.

Red-teaming and bounty programs - In the spirit of getting ahead of attackers, red teaming is used to discover vulnerabilities before they are leveraged by bad actors. Many organizations also offer bug bounties to white-hat hackers, often working independently, as a means to incentivize the discovery and disclosure of vulnerabilities.

Take-away

When it comes to cybersecurity, the task is all-encompassing, never-ending, and deeply intertwined with human behaviour and tendencies. When you bring an AI tool into your organization or integrate an AI function into your product, how are you addressing the potential vulnerabilities being introduced?

This is part 3 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Model inference accuracy can be a matter of life and death - but is it so simple to assess?

Accuracy is the degree to which a measurement or prediction conforms to the known, correct value. Machine learning models are used in countless ways for both classification and prediction. In these functions, model inference accuracy refers to how well the models perform when making classifications or predictions on new, unseen data. It is a measure of the model’s reliability and effectiveness in real-world applications.

Why It Matters

Inaccurate inferences can cost an organisation money, break related processes and even harm humans. The models are very complex, such that visibility to justify an inference is usually not possible. While models may provide a confidence interval along with their predictions, wrong answers provided with high confidence are very dangerous. Accuracy metrics should be tracked at every stage of the development and deployment pipeline.

Real-World Example

Google Flu Trends promised to turn unequaled access to multi-spectral data into formidably accurate flu predictions. But in the end, it could be outperformed by simple algorithms that only used current and past flu rates. This example shows that more data did not necessarily lead to better predictions.

Key Dimensions

Expected value: accuracy as a number doesn’t tell the whole story. The cost of a bad prediction vs. the expected value of a good prediction matters.

Context matters: the type of classification or prediction task being performed and real-world characteristics of the data (e.g. rare event) will dictate what metrics should be used to measure accuracy.

Garbage in - garbage out: this old idiom holds true across the entire machine learning pipeline, from the training data and labels used, to the real world data on which inferences are made.

Given the importance of accuracy in every implementation of machine learning, how should you measure it? Are you tracking it appropriately? What safeguards are in place to guard against failure? If these safeguards involve humans, are you aware of the new layer of complexity you’re introducing?

This is part 2 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

If you know, you know. And until then, you don’t.

Algospeak is coded language or modified terms, often with intentional misspellings, meant to discuss a sensitive topic online while evading detection by content moderation algorithms.

Why It Matters

Coded language is nothing new, but its need and use is being shaped by technology. Content is ranked and many serious topics are tightly controlled due to an impressionable audience and the inability of moderation algorithms to distinguish between a serious but constructive discussion and nefarious content that may glorify or otherwise inappropriately discuss the topic.

Real-World Example

In January 2025, “cute winter boots” was a term trending on TikTok and other platforms. One may initially think this refers to a seasonal item many of us need. In actuality, it references the wave of raids carried out by the U.S. Immigration and Customs Enforcement on sanctuary cities. The acronym for this agency (ICE) is the apparent connection to the term, perhaps aided by the coincident seasonality.

Key Dimensions

Speed of trends: new terms that trend quickly can be favoured by ranking algorithms, which goes beyond the initial purpose of evading moderation. The existence of influencers with large followings helps speed up the trend.

Ethical dilemma: Platforms are to balance free speech with a need to curb harmful content.

Context collapse: Users unaware of the algospeak terms may misunderstand the intended meaning. Large language models need to be retrained or fine-tuned to include the new context that’s been created and may be short-lived.

Hijacking: Algospeak terms that are trending may hijack or overshadow the conventional meaning of the term.

Is algospeak only a concern for large social media platforms or users of these platforms? I don’t think so. Large companies have grown the use of their chat-based tools (Teams, Slack, etc) and while employees may seek to cloak topics for different reasons than TikTok users, these companies may see their monitoring tools react to coded language in unexpected ways.

This is part 1 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Alarm fatigue is the effect by which humans become desensitized to notifications and alerts due to their high frequency. This effect is worsened by alarms that are too sensitive or simply non-actionable.

Why It Matters

Alarm fatigue causes alarms to be ignored while still potentially being a distraction. Humans fail to take appropriate action, even in high stake environments. This problem is widely recognized in healthcare where studies have shown that there are hundreds of alarms per hospital bed per day. To make matters worse, there is no standardisation of tones or sound patterns to indicate the severity of the alarm or even the source.

Alarm fatigue matters outside of healthcare too. Every alert a system or software produces consumes some of the cognitive bandwidth available to a human to do a task, or can serve as a distraction. Alerts that are ignored or even defeated also provide subtle information to the user as to the operation profile of the system.

A Real-World Example

Tesla’s Autopilot system is known for “nagging”, which is a prompt to force the user to interact with the car while it is driving autonomously. The internet is full of complaints from owners unhappy about the frequency of these alerts. It is also full of examples of people who learn how to subconsciously satisfy the alert by interacting with the controls, or even commercial products such as steering wheel weights to physically defeat the alert.

Plainly, this is because most of the alerts are unactionable.

Another important factor is that the nagging changes with software updates, so the system as a whole may change from one drive to the next. It is unclear what role the various implementations have played in crashes to date, but the NHTSA has mandated changes to Tesla’s software.

Key Dimensions

Here are some key things to consider - but it’s far from a complete list.

Frequency of actionable alerts: what percentage of alerts provide zero (or negative) value?

Safety profile of the environment: are alerts escalated according to the risks involved in a way that’s easily understandable by humans?

Human situational awareness: are the users instinctively aware of both what they are ignoring and paying attention to as it relates to the complex environment the alerts relate to?

There is already debate over the problem of overly anthropomorphizing physical robots. But can the same debate be had with text-generating models? Absolutely.

Upon its release, we were collectively and immediately impressed by ChatGPT. Our amazement was fuelled by the model’s ability to carry a conversation like a real person. In short, it passed the Turing test.

The Turing test is a test of anthropomorphization. By presenting all output as complete written prose, ChatGPT poses as a person instead of software. Until ChatGPT, none of us had received such responses from anything other than a human.

This is problematic. The use of natural language implies intelligence. Humans are predisposed to be fooled by this, to varying degrees. The use of natural, conversational and friendly language skews the expectations of the person interacting with the chatbot. They are likely to ascribe qualities to this model which are equivalent to those of a human producing the same language.

OpenAI knows this. They know the same model was much less impressive before being packaged as a conversational chatbot. This may be why they are continuing down this path with GPT-4o, one of openAI’s latest flagship models. In fact, they are doubling down on this theme.

There are two aspects I’ll mention. The first is chain-of-thought reasoning. It was quickly observed with the original ChatGPT that prompting it to “think step by step” often yielded responses of higher quality. Many assume the higher quality responses are the result of the model being coaxed to “think” more carefully or thoroughly. But there’s a simpler explanation. By using such language in the prompt, the model’s output is seeded more heavily towards the training data which consists of responses provided in such a stepwise fashion. These tend to be of higher quality, for reasons which would be interesting to explore further.

OpenAI has now baked in the chain-of-thought prompting into the model’s output. The model is very verbose in this way. The output will consist of a plan with some amount of recursion to execute this plan. This is a nice trick. This lends further illusions of human-like behaviour, in tandem with the previous conversational aspects.

The second misleading problem is the display of “thinking time” as a header to the output. A ChatGPT session powered by the GPT4-01 model will inform you that it “thought” for a set amount of seconds. In terms of user experience, this is a design decision which has nothing to do with the model. The user is left to assume this thinking is analogous to human thinking.

I’ve seen one user exclaim: “Imagine if you let it think for 6 months!”. Anecdotally, there appears to be no correlation between thinking time and output quality. In fact, long thinking times appear to be indicative of a time-out mechanism on the model not reaching a desired output. Perhaps this is due to the recursion imbued by the chain-of-thought prompting mentioned previously. If that’s the case, it’s interesting to observe that these two design decisions are interfering with each other in unexpected ways.

It’s clear that OpenAI is making design decisions with the purpose to lend humanness to their models. These are design decisions made to influence how users interact with and perceive their products. In the meantime, humans are predisposed to conceive human-like qualities of objects, machinery and animals. Or in this case, mathematical models.

But what Im noticing is that those pushing the hardest for the adoption of New AI know the least about the Old Lessons.

We’re being told to quickly get with the times and build with New AI, lest we be left completely behind and irrelevant.

But what I’m noticing is that those pushing the hardest for the adoption of New AI know the least about the Old Lessons.

I’ll never pretend that I can write-up or even summarise the Old Lessons in a LinkedIn post. They are based on the life work of countless individuals and span dozens, if not hundreds, of books.

But just knowing these Old Lessons exist is a massive head start - and puts you ahead of most.

The Old Lessons came from real-world applications built within a particular domain or industry. Many Old Lessons remained industry specific for a long time, though guidelines often emerge to generalise them more broadly. The modern office chair evolved from fighter jets.

Safety critical and automation heavy domains are awash with Old Lessons: aviation, industrial automation, medicine, high-rise construction, etc.

There are many academic domains that underpin Old Lessons. Psychology, cognitive science, human factors, economics, etc.

Sadly, ignorance of Old Lessons is pervasive in the tech industry. There is perhaps no better example of this than the Head of Autopilot Software at Tesla stating complete ignorance of basic human factors and human-computer interaction concepts during a legal deposition.

Not adopting new technology may set you back. Ignoring Old Lessons is guaranteed to do so.

Choose any of the many ways algorithms have steered a product off the rails recently. I’m happy to provide some examples if you need one to choose from:

Amazon uses an AI recruitment tool that shows clear bias against women.

Uber self-driving car fatally strikes pedestrian despite test-driver at the wheel.

YouTube Kids autoplays content toward a path of questionable rabbit holes.

Apple credit algorithm sets limits based on gender instead of credit worthiness.

COMPAS judicial recidivism tool presents bias against Black defendants.

Instagram’s automated content curation has proven to be unhealthy for the mental health of teenagers, particularly girls.

Tesla’s autopilot system implicated in multiple crashes where motorcyclists are hit from behind despite a driver being at the wheel.

Now here comes the experiment. Run your preferred AI governance or enablement framework against any such scenario.

Would these problems have been prevented?

Would they have been detected early?

How would your organisation have weighed the lapses in safety and ethics against business objectives if the algorithm had been providing value?

Each one of the above was a public relations headache for those involved. Most of these issues were raised by the public, but only after great harm was inflicted.

Is there a single framework that would guard a company, its employees and the users of its products against such future harms - and those yet to be imagined.

Are you being let down by AI agents or other LLM-based technologies that promise the ultimate efficiency gains by reducing your reliance on humans? Are you finding that these systems cause your humans to check out mentally, rendering them ineffective for those critical times that require oversight?

Well fear not, because I’m here to tell you about this one dirty little secret from the decades-old world of industrial automation: participatory placation.

You see, people naturally disengage from tasks in which they are not active participants, even if they are meant to supervise or oversee them. This renders them unaware of the dynamic state of these processes. That is, they have little to no situational awareness and lack the ability to cooperate with an automated system.

The solution is to create tokenized touch points that keep a user engaged. Automation is still used to remove inefficiencies, but inconsequential but satisfying tasks are assigned to the human to keep them engaged. That way, when failures occur, or the system finds itself outside its scope of operation, a willing and informed human is available to take over.

The tone of this post is tongue-in-cheek. But the lesson remains. Decades have been spent addressing automation problems before the advent of AI. Don’t implement AI without learning from these lessons first..

Human factored AI is the thoughtful integration of human factors to AI-enabled systems

First, what are human-factors?

Human factors (or human factors engineering) pertains to optimising the completion of tasks by people or organisations, taking into account the tools and systems used to perform these tasks. A primary goal of human factors is to optimise both human well-being and task performance.

The results of human factors are all around us. Ergonomic office chairs, door handles, pedestrian crosswalks, vending machines, etc. In some ways, it’s more readily apparent in the physical world, where good human factors ensure ease of use and safety. However, it is also pervasive in the digital world, such as your email client, the interface of an ATM or most smartphone applications. Human factors also impact how we transition from an older technology to a newer one.

How is human-factored AI different?

Human-factored AI is the application of human factors to AI-enabled systems. There are two important traits that warrant a distinction from traditional human factors

1. The speed of development and distribution of AI systems.
2. The non-deterministic behaviours and output of AI systems.

Isn’t that just human-centred AI?

Not really. The term human-centred AI has been diluted to great extent in recent years, and any specific meaning has since been eroded. Even if it had maintained a more precise meaning, the term implies the will and conscience of a user or human. However, human factors readily impact humans at a subliminal or imperceptible level.

So where do we go from here? I’m not sure, but I’m working on it.

In fact, most technologies that automate or enhance processes that involve humans contain “ironies of automation”. For example, an assembly line is created to automate a manufacturing process. However, when something goes wrong, the entire thing must come to a standstill. All production may need to immediately stop to resolve the issue. This is typically not the case when humans build things.

Another example could be automating monitoring software for a control room. The software may reduce the cognitive burden of the humans responsible for the control room. However, when something does go wrong, it may take significantly longer for a human to then gain the situational awareness needed to resolve the issue as they have been out of the loop for some time.

A third example are alarms meant to draw attention, such as those used by patient monitoring equipment in a hospital. At a certain threshold, such alarms may cause alarm fatigue, which leads to a decrease in a care provider’s response rate and an increase in their response time.

In information technology, we have long depended on computers for the retrieval of accurate information. When a database is queried, whatever was stored can be reliably extracted. Any errors are bound to have happened at insertion or retrieval. This is barring a bit flip or other soft errors, but again, there are safeguards against these. Humans exercise their judgement but depend on the reliability of information retrieved from computer systems, often in automated ways.

The irony is that through LLMs, we’re creating computer systems that generate information in inconsistent and unreliable ways, but presented with a veneer of confidence by way of grammatically correct and tone appropriate language.

We’ve created information technology that requires fact-checking.

In 1951, Paul Fitts published a list of 11 statements that allocated tasks to humans depending on their strengths and weaknesses. Despite criticisms, it has stood the test of time remarkably well.

Neural nets encroached on the functions of detection and to some extent, long term memory. LLMs seem to further erode the functions which had been allocated to humans so far. In particular, LLMs seem able to improvise. They also generate text with confident tones of judgement, induction and perception.

Is the output of LLMs robust, as it relates to these functions? Or is there a significant shortfall, hidden behind the veneer of formalised language output, ie. a stochastic parrot?

This post was originally authored as a post on LinkedIn.