I watched the last episode last night and again feeling that the Superintelligence chapter ‘The superintelligent will’ is a good companion. The question of ‘the relation between intelligence and motivation’ feels like the crux of our fictional imagination of what superintelligence will be like and the true horror of potential realities.
I’ve mentioned it before, but again, well worth a read is Tim Urban’s short story of Turry, a fictional handwriting robot arm and AI, that has been instructed to “Write and test as many notes as you can, as quickly as you can, and continue to learn new ways to improve your accuracy and efficiency.” Search this page for Robotica to find the story.
Rather than just doom and gloom or sci-fi fanboyism, my interest in this area is slightly more focused on what these theories and thought experiments offer to understanding human actions and motivations in the here and now. A similar feeling as when I read How the Mind Works and The Man Who Mistook His Wife For a Hat. These slightly pop-science books should be part of any designers reading list I think. It’s not enough to learn what you’re told good design is, without understanding more about the mechanisms of the organisms that agree or disagree with you.
Documenteering 