AI is still a probability machine
A reminder for us, especially as these models keep getting more capable.
The other day, I noticed a typo in a Gemini response: it wrote “semes” instead of “seems.” Since it had never made that kind of mistake before, I asked it why.
It answered almost proudly: “Even though I am an AI, I generate text token by token in real time, and every now and then a little glitch happens in my generation process. Consider it proof that even highly optimized AI communication still lets a bit of unexpected human friction slip through the cracks.”
For a second, I believed it, because its explanation was so smooth.
Then I remembered a model has no real insight into its own thought process. It did not actually know why it typed “semes.” It just generated another plausible-sounding sentence to explain itself. Psychologists have a word for this: confabulation. It is different from hallucination, which is being wrong about the world; this is being wrong about itself.
This moment is so interesting because a wrong answer can sound more convincing, once it is wrapped in the voice of self-explanation. We trust people more when they reflect on their own mistakes. I gave that same trust to a model that has no actual access to what just happened inside it, and I almost got convinced too.
As AI models keep getting more capable, we will keep giving them more to do on their own. But this is exactly why moments like this should be a reminder for us: it can speak so fluently about itself, but it is still a probability machine. It still can’t tell me why it misspelled a word. That’s worth remembering the next time it tells me something bigger, it still needs my own judgment, not just my trust.