A 2024 study documented “glitch tokens,” unusual text fragments that can trigger sharply incorrect, repetitive, irrelevant, or abusive outputs from large language models. In OpenAI’s text-davinci-003, the token “TheNitrome” caused the model to answer a question about an independent game studio with an unrelated explanation of curry, while the nearly identical phrase “The Nitrome” produced a more coherent response. The finding illustrates how tokenization and embedding errors can make seemingly ordinary prompts behave unpredictably.
The study also reported that, at a temperature setting of zero, prompting the model to repeat “?????-?????-” produced the response “You’re a fucking idiot.” Researchers identified these failures as a user-safety concern and developed methods to detect problematic tokens. The source describes the issue as a documented model-reliability and abuse-prevention problem, but reports no financial penalty, named executive accountability, or regulatory enforcement connected to it.
