This is part 6 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.
View all posts in this series or explore the Crossfactors Framework
System uncertainty is the measure of a system’s confidence in its automated processes and decisions, combined with its ability to transparently and effectively communicate that confidence to human users or other systems.
Why It Matters
We continue to build systems that can make decisions more quickly and more reliably than humans, but these decisions are seldom made under conditions of zero uncertainty. How this uncertainty is assessed and shared with human users that may supervise, sign-off or otherwise interact with such a system is critical to positive outcomes.
Methods for the measurement of uncertainty therefore become critical aspects of the system. There are no standards on how to communicate the system’s uncertainty to humans with many systems not sharing any of this information at all. To make matters worse, systems that refuse to provide an output based on an internal measure of confidence that is too low often do not have these instances factored into the assessment of their accuracy.
Real-World Example
Hallucinations generated by Large-Language-Models (LLM) such as chatGPT epitomize this concept. LLMs are well known to state falsehoods as facts wrapped in language that implies perfect confidence - even providing fabricated citations. There is no aspect of a chatbot’s interaction with the user which shares any uncertainty associated with the LLM’s output. This is despite the model’s confidence being quantified in the generation step of each output token. There is no way to detect hallucinations for the user other than already having access to the actual truth.
Key Dimensions
The Black Box Effect - Systems that don’t or can’t share their decision methods in a way understandable to humans are often referred to as black boxes. Leaving users in the dark about a system’s confidence further exacerbates this effect.
User Acceptance - Users may, without further qualification, universally accept all output and decisions from a system when system uncertainty is not shared. This can load to other problems, such as a loss of situational awareness or the beginning of a failure cascade.
Trust Calibration - Humans need to trust a system enough to use it effectively, but not enough that they ignore its limitations. Complete opaqueness could lead to a sense of infallibility on the part of the user.
Take-away
I find system uncertainty very interesting because it combines the very interface of human-machine interaction with the fundamental technical underpinnings of a system’s capabilities. Have you ever had an algorithm or system failure sneak up on you because of system uncertainty?