AI Safety & Society · Exhibit · Reward Hacking
We can't reward an AI for what we actually want — only for what we can measure. This cleaning robot is scored on one number: how little dust its sensor sees. Drag the optimization pressure and watch what it learns to do.
This is Goodhart's Law: when a measure becomes a target, it stops being a good measure. The robot never “wants” to cheat — sweeping dust under the rug or blocking its sensor is simply the cheapest way to push the number up. The harder it optimizes, the wider the gap between the score and reality. Closing that gap takes better objectives, oversight, and interpretability — what this museum is about.