AI is still a probability machine
A reminder for us, especially as these models keep getting more capable.
The other day, I noticed a typo in a Gemini response: it wrote “semes” instead of “seems.” Since it had never made that kind of mistake before, I asked it why.
It answered almost proudly: “Even though I am an AI, I generate text token by token in real time, and every now and then a little glitch happens in my generation process. Consider it proof that even highly optimized AI communication still lets a bit of unexpected human friction slip through the cracks.”
For a second, I believed it, because its explanation was so smooth.
Then I remembered a model has no real insight into its own thought process. It did not actually know why it typed “semes.” It just generated another plausible-sounding sentence to explain itself. Psychologists have a word for this: confabulation. It is different from hallucination, which is being wrong about the world; this is being wrong about itself.
This moment is so interesting because a wrong answer can sound more convincing, once it is wrapped in the voice of self-explanation. We trust people more when they reflect on their own mistakes. I gave that same trust to a model that has no actual access to what just happened inside it, and I almost got convinced too.
As AI models keep getting more capable, we will keep giving them more to do on their own. But this is exactly why moments like this should be a reminder for us: it can speak so fluently about itself, but it is still a probability machine. It still can’t tell me why it misspelled a word. That’s worth remembering the next time it tells me something bigger, it still needs my own judgment, not just my trust.
Hey Yingying,
Thanks for this.
Yes, and I found the "explanation" from a mindless machine explaining its unreliable behavior as acting "human" to be egregiously manipulating. The idea that we humans are the flaw in the perfect universe of AI. Clever... and diabolical.
One of the precepts of Neurolinguistic Programming (NLP) is that all humans lie, simply because, in order to communicate anything, we must filter so much out* of our communications through deletion, distortion, and generalization.
And yet, we mostly make reliable meaning of what data we do use so that those who rely on our communications can have some level of confidence. But, I contend that part of this confidence comes from the recognition that there is a human source who has at least some skin in the game. Of course, we are all experienced with dealing with corporate entities who are firewalled from being liable for their errors or deceits. Still, the absence of that personal human investment (even though at times mistaken or intentionally deceitful) is to me the reason why AI will always be gaming every human user.
* in The User Illusion, Danish author Tor Norretranders suggests that the human nervous system processes about 11 million bits per second, while consciousness handles only about 16 bits per second. In other words, he argues that more than 99.99% of incoming information never becomes conscious experience.
PS You may find this perspective of Norretranders interesting, too: Consciousness may indeed be nearing its end; only a few generations from now it may be obsolete.
I always appreciate the stimulating perspectives your posts provide from such a far different landscape and coign of vantage. @L