This is part 24 of “101 Ways AI Can Go Wrong” - a series exploring the interaction of AI and human endeavor through the lens of the Crossfactors framework.
View all posts in this series or explore the Crossfactors Framework
Hey! What’s going on!?
Todays’ factor is Situational Awareness and it’s #24 in my series on 101 Ways to Screw Things Up with AI.
It’s a big one.
What is it?
Situational awareness refers to the process of gaining and maintaining awareness of the dynamic processes involved in a task. This includes perceiving, understanding and predicting the state of a system or environment in real time.
Why It Matters
Traditionally, situational awareness was judged from the human’s perspective when performing a task in a changing environment, with other humans and/or using automation. But today, situational awareness needs to include the perspective from the system since AI systems are opaque and make decisions using non-deterministic mechanisms. The situational awareness of humans and AI systems are likely to be completely different, even for the same task - and the interplay between them is likely to be unpredictable and give rise to unexpected outcomes.
Real-World Example
Let’s take a very simple example to underscore this concept.
A Tesla vehicle in self-driving mode misjudges the relative speed to a vehicle ahead, due to sensor limitations or misclassification. Its situational awareness is inadequate and doesn’t accurately represent the current environment. Meanwhile, the driver has not had to take the car’s controls for several minutes and though they are looking ahead, they aren’t cognitively engaged enough to judge the relative speed for themselves. The driver in this case lacks situational awareness of the environment (the vehicle ahead) and of the automated system (which knows about the vehicle but is misreading its speed).
Does the vehicle know that the driver isn’t ready to takeover? Well, that also depends on the system’s situational awareness!
This entire situation, which I would mention is happening on our roads everyday, will come to a head by a collision, a close call, abrupt braking or an alert to the driver - at which point the takeover problem comes into play.
Key Dimensions
Dynamics - situational awareness is not a snapshot but a continuous process.
Perception - gathering and elevating the correct information, for either humans or AI systems, is a prerequisite.
Comprehension - combining relevant information to create an understanding and meaning. These are very different processes for humans vs. machines.
Projection - critical for most tasks is an ability to infer the future state of a system, whether an outcome will be achieved and if the current course must be corrected.
Take-away
Maintaining situational awareness is your safeguard against failure.