Michael Lewis warned that baseball scouts were heading for extinction. 15 years later, MLB teams had hired 32% more of them.

Geoffrey Hinton said we should stop training radiologists.

The Mayo Clinic now has 55% more of them.

Those touting the technology are quick to make predictions about people. But the technology doesn’t decide what happens. How people use it does. To predict that, you need to understand two things.

First, the job. The inputs and outputs, the edge cases, the workflows, the incentives.

Second, and often overlooked, the cognitive predispositions of the people doing it.

When I was building AI for radiologists, the industry was focused on one kind of error: missing something on the image. But radiologists make at least a dozen others. They anchor on the clinical history. They stop looking after the first finding. Put an AI flag on the screen and they may trust it too much, or after enough false alarms, not enough.

Baseball learned the same lesson. Teams had more data than ever, but a hitter can only hold a thought or two in the batter’s box. The edge went to the coaches who knew which one to give him.

The technology didn’t replace those people. It ran into how they think.

Now predictions abound about the rest of us. AI will replace knowledge workers, we’re told, starting with entry-level jobs.

I don’t think the Big AI labs understand the job.

Their predictions are making the same mistake: they’re looking at what the technology can do, not at how people will work with it.

Yesterday I argued that AI will make human attention and judgement the scarcest resource in enterprise. Baseball already ran that experiment.

The first change is the one everybody knows. Billy Beane had one of the smallest budgets in the league. Payroll was the scarce resource he needed to allocate. He used data to find the players the market was mispricing because other team’s scouting departments made decisions on intuition and feel. He used this edge to make the playoffs four straight years.

This edge was soon eroded. Before the book even came out, Oakland’s front office staff were recruited by other teams. By 2004, teams were paying full price for on-base skill as quantified by analytics. Eventually, every team had an analytics department. The data, the models, the analysts all became commodities.

Data got cheaper and more plentiful. The stadiums were equipped to track everything that could be tracked. Today, every pitch, every swing, every spin rate is tracked and shared with all 30 teams. But a hitter can’t think about forty metrics in the batter’s box.

That’s when the second change began.

Teams recognized the competitive edge had quickly moved. Every team had an abundance of commoditized information and established analytical methods. The advantage became a team of humans capable of effectively distilling and executing on this information. The bottleneck moved from information to human attention and judgement.

So teams did something counterintuitive: they hired more people. Some teams now carry three or four hitting coaches, and their job is to translate the data into something a player will trust and act on. A manager can only have so many conversations per day. At every level, for every matchup, it’s someone’s job to only pass along the insight that matters.

That’s the version of the Jevons Paradox I pitched yesterday, playing out on the diamond. Cheaper data didn’t reduce the need for human judgement. It multiplied it.

Every company is about to go through both changes. But most may only be looking at the first. Building an advantage there will be short-lived. The second is the one that decides who wins.

If enterprise success has always come down to allocating scarce resources efficiently and effectively.

and if AI is making a whole class of once-scarce resources, such as analysis, code, content and expertise nearly free at the margin;

and if access to non-frontier models is becoming a commodity;

and if your competitors have access to the same intelligence as you do;

and if the same models, in two different companies, will only product different results depending on who’s asking the questions and who’s judging the answers;

and if the cost of AI output continues continues falling while demand for inference continues to rise (the mainstream view on Jevons Paradox);

and if more analyses, more drafts, more code, more agents are doing more things;

then human attention and judgement become both your only differentiator and your scarcest resource. This is the real Jevons Paradox.

The job of management hasn’t changed. It’s still resource allocation. The winners in the AI era won’t be the companies with the best models or fewest employees. They’ll be the ones that figure out how to deploy human attention and judgement most effectively.

Every email has a cost.

It costs something to generate, distribute, and store emails. These costs are understood and easily measured.

The cost of the email server is trivial. That was never the real cost of email.

The real cost is the cost of reading it, assessing it, acting on it, or choosing not to act on it.

This is the cost of human attention. There’s no dashboard for it.

Some organizations figured email out. Not by buying better servers. They did it by building discipline around the human side. Norms were set about when to send or forward and when not to. Rules about who actually needs to be copied. Cultural permission not to respond to everything instantly.

They managed the expensive, human component instead of optimizing the cheap part.

Right now, everyone is measuring AI by its IT costs. Tokens, compute, latency. The numbers on invoices and dashboards. But the organizations that will win with AI will be those that build discipline around the human and team elements.

When servers are cheap, it’s easy to let email bury your people.

The real cost of AI will never be the inference bill. It will be the cost of human attention, collaboration and judgment. These are the things no dashboard measures and no models replace.

Your team is juggling context windows.
Worrying about compaction.
Switching models.
A/B testing prompts.
Trying to keep track of what tools are being called.

That is not AI’s promise, nor its potential.

In the early days of computing (and I do mean early), moving text between applications was a dark art. Many simple operations placed a heavy burden on the user. You had to remember a command, commit to an action and then specify what it applied to. On some systems, this required the user to have intricate knowledge of the system’s memory. Error recovery was brutal. Sound familiar?

Then copy and paste was invented. Select the text first. Then decide what to do with it. Paste it once, twice or never. Let the software manage the memory implications, not the user.

Your people have a task in mind. But they are forced to think about their prompt, its style, the context, the model, the availability of tools and plugins. The intent is in their head. The interface is demanding something else entirely.

We are in the modal editor phase of AI interaction. Powerful, but asking too much of the humans using it. And squandering bandwidth that could be used to add actual human intelligence to their work.

AI will have its copy and paste moment. The question is whether your organization is building workflows around the current friction or preparing for what replaces it.

They say that those who don’t learn how to use AI will be replaced by those that do. But few are seeing ahead to what this implies.

If most business decisions are bounced of chat bots like ChatGPT, Claude, CoPilot and Gemini, what happens?

The responses will all converge to some maximally probable output.

They will be average.

They will be unremarkable.

What makes this worse is that the models have their own incentives.

They’re optimized to satisfy you.

To bring you back. To tell you what you want to hear.

The same engagement mechanics you hate in your Instagram feed are baked into that chat window.

The thing no one mentions about Moneyball is that it quickly stopped being a competitive advantage once everyone adopted it. The advantage moved to identifying where and why Moneyball was wrong.

Do you really want to bake average thinking into your company’s culture?

Now is the time to build a competitive advantage using AI, but only if you identify its shortcomings.

This has rendered email inboxes largely ignorable. the bulk of it goes to spam. The rest is filtered. What remains is quickly skimmed over. Most emails that don’t compel an immediate response will likely be immediately forgotten - or deferred until it is. Unless a sender has earned a place on the recipient’s white list, whether explicitly or implicitly, most emails will never be seen.

Information feeds taken over by AI-generated content will soon meet the same fate. Short and long form content, with and without images, can be generated at very little cost and in very little time. it can be topically relevant, context-relevant while fooling most of us as to its provenance. There is so much of it in our online information feeds, including here on LinkedIn, that it is mostly ignored.

Email is still used everyday. It’s critical both at the individual and enterprise levels. But to the recipient, the first question is always “who is the sender?”

As we browse our LinkedIn feeds, we all ask ourselves a version of that question. Who is behind this content. but not just who posted it - who created it. Did they create it themselves, use a ghost-writing service, chatGPT or some other LLM? Unconsciously (or consciously for some!), we look for any reason to ignore it.

But if the content is good and the creator is genuine, then maybe it’s worth our time. Unfortunately, the cues we seek are hard to find.

This is part 24 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Hey! What’s going on!?

Todays’ factor is Situational Awareness and it’s #24 in my series on 101 Ways to Screw Things Up with AI.

It’s a big one.

What is it?

Situational awareness refers to the process of gaining and maintaining awareness of the dynamic processes involved in a task. This includes perceiving, understanding and predicting the state of a system or environment in real time.

Why It Matters

Traditionally, situational awareness was judged from the human’s perspective when performing a task in a changing environment, with other humans and/or using automation. But today, situational awareness needs to include the perspective from the system since AI systems are opaque and make decisions using non-deterministic mechanisms. The situational awareness of humans and AI systems are likely to be completely different, even for the same task - and the interplay between them is likely to be unpredictable and give rise to unexpected outcomes.

Real-World Example

Let’s take a very simple example to underscore this concept.

A Tesla vehicle in self-driving mode misjudges the relative speed to a vehicle ahead, due to sensor limitations or misclassification. Its situational awareness is inadequate and doesn’t accurately represent the current environment. Meanwhile, the driver has not had to take the car’s controls for several minutes and though they are looking ahead, they aren’t cognitively engaged enough to judge the relative speed for themselves. The driver in this case lacks situational awareness of the environment (the vehicle ahead) and of the automated system (which knows about the vehicle but is misreading its speed).

Does the vehicle know that the driver isn’t ready to takeover? Well, that also depends on the system’s situational awareness!

This entire situation, which I would mention is happening on our roads everyday, will come to a head by a collision, a close call, abrupt braking or an alert to the driver - at which point the takeover problem comes into play.

Key Dimensions

Dynamics - situational awareness is not a snapshot but a continuous process.

Perception - gathering and elevating the correct information, for either humans or AI systems, is a prerequisite.

Comprehension - combining relevant information to create an understanding and meaning. These are very different processes for humans vs. machines.

Projection - critical for most tasks is an ability to infer the future state of a system, whether an outcome will be achieved and if the current course must be corrected.

Take-away

Maintaining situational awareness is your safeguard against failure.

This is part 1 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.

View all posts in this series or explore the Crossfactors Framework

Alarm fatigue is the effect by which humans become desensitized to notifications and alerts due to their high frequency. This effect is worsened by alarms that are too sensitive or simply non-actionable.

Why It Matters

Alarm fatigue causes alarms to be ignored while still potentially being a distraction. Humans fail to take appropriate action, even in high stake environments. This problem is widely recognized in healthcare where studies have shown that there are hundreds of alarms per hospital bed per day. To make matters worse, there is no standardisation of tones or sound patterns to indicate the severity of the alarm or even the source.

Alarm fatigue matters outside of healthcare too. Every alert a system or software produces consumes some of the cognitive bandwidth available to a human to do a task, or can serve as a distraction. Alerts that are ignored or even defeated also provide subtle information to the user as to the operation profile of the system.

A Real-World Example

Tesla’s Autopilot system is known for “nagging”, which is a prompt to force the user to interact with the car while it is driving autonomously. The internet is full of complaints from owners unhappy about the frequency of these alerts. It is also full of examples of people who learn how to subconsciously satisfy the alert by interacting with the controls, or even commercial products such as steering wheel weights to physically defeat the alert.

Plainly, this is because most of the alerts are unactionable.

Another important factor is that the nagging changes with software updates, so the system as a whole may change from one drive to the next. It is unclear what role the various implementations have played in crashes to date, but the NHTSA has mandated changes to Tesla’s software.

Key Dimensions

Here are some key things to consider - but it’s far from a complete list.

Frequency of actionable alerts: what percentage of alerts provide zero (or negative) value?

Safety profile of the environment: are alerts escalated according to the risks involved in a way that’s easily understandable by humans?

Human situational awareness: are the users instinctively aware of both what they are ignoring and paying attention to as it relates to the complex environment the alerts relate to?