One part of the architecture of the new Defunctland video that I really liked was how the inclusion of all the sketchy interactions with "living characters" (sexual harassment, transphobia, etc) wound up tying in well with the concluding thesis statement abt how theme park attractions are all made and operated by real people somewhere in there. You certainly only get pushy "flirting" from theme park characters when there's a real person behind it who feels like they've got enough anonymity and plausible deniability of "playing a character" to get away with it. And these human biases are built into the soundboard characters too: voice line options are sorted by whether a targeted guest is a boy or a girl (or adult man or woman), and the options that follow that choice are based on social presumptions for said gender. Roz has "flirtatious" voice line options for adult men, for example, but I doubt operators are given those same options if the targeted guest is a woman. Presumably soundboard characters have many more princess-related options under the "girl" menu, and this is based off assumptions the real people putting the attraction together made. And in operating the "living character", the real person working the soundboard is making choices based on thier own biases and assumptions about a given guest.
So of course automation is dreamt up to be a solution to the "embarrassment" of failed human interactions and subsequent poor customer experiences. We wouldn't want another Stitch gender rant. But behind all the attempts at automating "living characters" are real human developers who are bringing their real human biases to their work. There are assumptions being made about what a given guest wants to hear, and what their human reactions to an automated character imply. Developers can try and assign quantitative values to a facial expression, but they're still making assumptions about how the numbers they come up with will apply to hundreds of thousands of people. Thus any time an automated character expresses social biases, it remains a reflection of its human "operators" despite the superficial removal of humans from the equation. (I'm also considering things like the orientalist tropes wrapped up in Destini's character, for example, to be an expression of social biases. Real people decided this was the best way to present the idea they've been developing to the public.)
I think this is all important to keep in mind as AI chatbots and facial recognition technology built on these same technical principles become more prevalent in broader society. These technologies are not exempt from human social biases because they were developed and programmed by real human beings. When automated facial recognition programs return false positives in criminal databases for a random Black person, or identify a US citizen as an illegal immigrant, that's not the cold, impartial thinking of a machine, it's the result of racism that came with the technology's development, and the decision by real people to use the technology to identify "real" criminals and illegal immigrants. We all had a laugh at that one TERF dating app that used facial recognition on applicants that ended up rejecting cis women it assumed were trans, but that isn't a result of those women not "passing" by the neutral logic of a machine, it's the result of the very same bigotry the app's creators baked into the system. At the end of the day, these technologies are just as human as anything else a real person creates, but they provide enough distance and plausible deniability from their developers that people feel comfortable using them as impartial parties passing logical judgement outside of human bias.
