Rotten to the core

I've been exploring the limitations of ChatGPT and other language models by testing their ability to handle simple transformations like the ROT13 cipher. The results are revealing about how these AI systems actually work.

When I asked ChatGPT to encode and decode text using ROT13, it consistently struggled with the letter-level transformations. This isn't a bug - it's a fundamental feature of how these models operate.

The Token Pool Reality

What ChatGPT does is literally just picking words from the pool of words it learned and arranging them in a way that they would most likely appear according to the context. It doesn't truly "understand" language in the way humans do.

The AI works through tokenization - breaking text into tokens and predicting the most probable next token based on patterns it learned during training. This token-based processing means the model doesn't truly understand individual letters or alphabets.

ROT13 Reveals the Limitations

When attempting ROT13 encoding, ChatGPT often produces gibberish text or repeats similar phrases. This happens because:

• The model can't perform precise letter-by-letter transformations
• It's trying to predict likely word sequences rather than following algorithmic rules
• Token-based processing interferes with character-level operations

This limitation extends beyond simple ciphers. It reveals something fundamental about how language models operate - they're sophisticated pattern matchers, not rule-following systems.

Philosophical Implications

This approach to language processing - predicting likely sequences without true understanding - might not be so different from how humans process language much of the time. We often speak in patterns and common phrases, drawing from our own "pools" of learned language.

The difference is that humans can also step outside these patterns when needed, applying rules and understanding context in ways that current AI systems cannot.

Understanding these limitations is crucial as we integrate AI systems into more areas of our work and lives. They're powerful tools, but they're not thinking in the way we might assume they are.

All Posts