I tried to report a security issue responsibly, and an AI safety filter stopped me.
This is a human-in-the-loop story.
I was using Claude to review a company’s bug bounty rules when its guardrails blocked my message. To continue, I had two options: reword it, or switch models.
I’m not even mad. It’s the same problem I work on, from the other side.
Automated checks judge the surface. Sometimes harmful content passes because it looks fine. Sometimes good-faith requests get blocked because they look risky. You can’t tune that away, only choose which mistake you’d rather make.
But look at what happens next. The check handed the decision back to me, the human in the loop, and the only options it gave me were to change my input or change the system.
Nothing in that flow asks whether the check was right.
So people learn to reword. A check that’s too sensitive doesn’t just block good requests. It teaches everyone to work around it, and it never finds out it was wrong.
Whoever designs the check is making that choice for everyone who meets it. The question is whether they’re measuring both kinds of error.
Which mistake does your organization’s AI oversight make more often? And would you know?