Abstrakt
Imagine a powerful future cleaning robot tasked with keeping streets clean. Engineers may be tempted to augment its intelligence to make cleaning more efficient.
At first, this may lead to a shorter route through the city. However, given a large enough solution space, the engineers may soon find themselves surprised by less orthodox solutions - like the robot locking up partying teenagers who create mess at night.
The robot is still fulfilling the goal given to it - cleaning the city. The engineers just forgot to give it the plethora of other specifications that the robot should be fulfilling while working on its primary goal.
For powerful future AI systems, the range of dangers associated with such specification failures spans all the way to existential risks.