The moral crumple zone
When an automated system fails, blame lands on the nearest human, who often had no real control.
Madeleine Clare Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction, Engaging Science, Technology, and Society, volume 5
2019
Madeleine Clare Elish coined the term in 2019 for the way responsibility for an automated system's failure gets attributed to the human operator closest to it, whose actual control over the system's behavior may be small or none. The name borrows from the part of a car that absorbs the impact so the rest of the vehicle survives.
On the canvas it is the Authority cell scored solid because a name exists, on top of an Evidence cell that cannot show what that person could have seen, and a Mastery cell that never checked whether they were qualified to intervene. The canvas lists it as an anti-pattern because it is what human oversight becomes when it is asserted in prose and never drilled.
Which decisions may an agent make alone, and who is able to stop it?
What would you show an auditor, and can the system produce it without being asked?
Who is qualified to overrule the machine, and how would you prove it?
