Abstract

Granting artificial intelligences (AIs) moral status creates a set of ethical dilemmas stemming from our ability to control their preferences. As an illustrative example, if we could program an artificial system worthy of moral agency to experience suffering when encountering the color violet, then we appear to have created a scenario where we have a moral imperative to remove any violet objects from our surroundings. AIs in this situation can effectively “hijack” our moral landscape, altering the moral valence of arbitrary world states. In this work, we formalize the phenomenon of moral hijacking, analyze its implications from the perspective of several ethical systems, and propose recommendations for constraining novel preference creation in moral agents.