I'd recommend everyone look at the work these people are doing in this field: intelligence.org
The Machine Intelligence Research Institute (MIRI) is a nonprofit focused on preventing human extinction from artificial superintelligence.
intelligence.org
North Box Systems