I ran into something last week that unsettled me more than expected. I was testing whether a model could translate a passage using only its own reasoning, so I hid the source text behind a simple substitution cipher before handing it over — the idea being that if the model could still produce a faithful translation, it would have to be doing something closer to genuine cross-lingual reasoning than pattern completion. It decrypted the cipher and recited the verse anyway. Correctly. Immediately. The Bible is so memorized by frontier models that hiding the source text doesn’t work — the model just decrypts and recites it anyway.
That’s stranger than it first sounds. This wasn’t a model translating an encrypted string. It was a model recognizing that a scrambled string of characters was shaped like a passage it already had memorized verbatim, discarding the cipher as a distraction, and reciting from memory. That’s a different capability than translation, and it means a huge chunk of what looks like “the model translated this well” for biblical text is really “the model already knew this text in every language I asked for.” You can’t use Scripture to test whether a model can translate, because for Scripture specifically, the model usually doesn’t need to.
This is a genuine methodological problem, not just a fun fact. If I want to evaluate — or train — a system’s ability to reason from meaning to meaning rather than reciting a memorized target string, I need source material the model hasn’t swallowed whole. Obfuscating the surface form doesn’t work, as I just found out the hard way. What might actually work is changing what the “source” is in the first place.
Acting the scene instead of laundering the words
There’s a practice-level version of this same insight that I’ve been leaning on for a while, mostly with human translators and increasingly as an instruction to models: stop translating the words. Act the scene. Then say what just happened in your own language.
The reasoning is the same as in prioritizing semantics over structures: a word is not a translation unit, it’s a residue left behind by a meaning that already happened. Translating the residue is really just copying with extra steps — the failure mode the cipher experiment exposed at scale. A model (or a person) staring at a string of tokens finds the shortest path to an output resembling those tokens, and if it’s already memorized the “right answer,” the shortest path is recall, not comprehension. Acting the scene forces a detour around that shortcut. You have to know who is speaking, to whom, what they want, what’s at stake, before you’re allowed to say anything at all. Only then do you say what happened, in whatever language you’re working in.
The more rigorous version: don’t feed it the words at all
The practice-level fix is good discipline, but discipline is not architecture, and I’d rather not depend on a model or a translator “resisting the urge” to recall a memorized string. The more rigorous fix, and the one I actually want to build, is to stop handing the model literal source text as the thing to be translated at all.
Instead, represent the passage as a language-agnostic semantic description: pericopes broken down into nested units of who is speaking, to whom, and what is being communicated — speech acts nested inside speech acts, the way narrative and dialogue actually nest inside each other. This is the same move I described in the case for dynamic equivalence alignments — form → meaning → form instead of form → form — pushed one step further upstream. Instead of asking a model to align a translation back to source words after the fact, you never give it the source words as the object to reproduce in the first place. Its job becomes representing the meaning of a discourse unit faithfully, not pattern-matching against a corpus it already has memorized down to the verse numbers. If there’s no recognizable surface string to recite, “recite the memorized answer” stops being a viable shortcut, and the model actually has to do the thing you’re asking it to do. This is really an extension of the point I made in beyond word alignments: the word was never the right unit, and now I have a concrete reason — memorization contamination — to stop treating it as one even operationally.
Where this is honestly still rough
I don’t want to oversell this, because I’ve already tried a related methodology and it only sort of worked. I built a multimodal process I’ve been calling Familiarize → Internalize → Articulate — acting out a passage, drawing the scene, studying the characters and key terms — before ever producing a rendering, as a more embodied version of the “act the scene” instinct above. In practice it can shunt between layers of context in ways that are only sometimes helpful; moving between acting, drawing, and terminology study doesn’t reliably converge on a clearer semantic representation, it sometimes just adds noise. And the supporting resources for it — the character studies, the term glossaries, the scaffolding that’s supposed to make internalization tractable rather than vibes-based — are still unfinished.
So this isn’t a solved problem, it’s a diagnosis with a promising direction attached. The cipher experiment told me the failure mode is real and larger than I assumed. The semantic-description architecture is my best current guess at a fix that doesn’t rely on willpower. Whether nested speaker/addressee/content units are actually the right granularity, and whether a model can be made to represent meaning at that level reliably, is still very much open.