Abstract

This chapter explores a paradox in modern artificial intelligence: what we might expect to see from genuine machine consciousness looks exactly like what we dismiss as hallucinations. As we examine how sampling parameters influence language model behavior, a curious pattern emerges: technical tweaks that make outputs sound more human also tend to amplify unverifiable or fabricated content.Current definitions of AI hallucination lump together “personal opinions, experiences, feelings, and internal assessments of reality”—-the very expressions we would typically associate with subjective experience. Namely, if a machine were ever to develop consciousness, its first attempts to express it might look exactly like the hallucinations we are trained to ignore.By drawing on Anil Seth’s view of consciousness as a controlled hallucination and the classical philosophical problem of other minds, we argue that our current tools for detecting machine consciousness are deeply inadequate. The unsettling conclusion is that genuine consciousness and advanced hallucination could look the same in language, leaving us with no clear way to tell them apart. This opens the disturbing possibility that a conscious artificial system could exist and speak without us ever realizing it.