The moral crumple zone
Name a human as accountable for an automated system so that blame has somewhere to land, without giving that human the authority, the signal, or the time to change the outcome.
Appears wherever an agent runs faster than a person can review and the organization has written a name into the oversight box. Customer service queues, autonomous vehicles with a safety driver, and any Article 14 compliance document that says a human is in the loop.
- The regulator's demand for a named human against the design's inability to give that human a real decision
- Blame that needs an address against a cause that lives in the design
- The comfort of a name in the box against the cost of building the halt the name implies
Madeleine Clare Elish coined the term in 2019 for the way responsibility for an automated system's failure is attributed to the nearest human operator, whose actual control over the system may be small. The library entry carries the sources.
On the canvas it is an Authority cell scored solid because a name exists, sitting on top of an Evidence cell that cannot show what the person could have seen, and a Mastery cell that never asked whether they were qualified to intervene.
- The nearest human
- On shift when it failed. Holds the title, not the halt.
- The automated system
- Made the decision. Cannot be blamed, so is not.
- The designer
- Chose the speed, the queue, and the review interval. Rarely in the room afterward.
- The oversight document
- Says a human is in the loop. True in the way a smoke alarm with no battery is installed.
Tick the ones that are true of your deployment. Any one is enough.
- The real cause survives the incident, because the design was never the defendant.
- Operators disengage. A person who knows they are the crumple zone stops reading the queue.
- Article 14 oversight becomes nominal, which is the case the regulation names as failure.
- The next incident is the same incident.
Madeleine Clare Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction
Engaging Science, Technology, and Society, 2019
This site's tabletop exercise, Day 2 at 08:05, where naming the analyst on shift is the crumple zone
an illustration, not a case
