Large language models (LLMs) can sometimes learn the wrong lessons — a flaw that may make them less reliable in real-world applications, a new study from the Massachusetts Institute of Technology has found.
Researchers report that instead of using domain knowledge to answer questions, LLMs can fall back on grammatical patterns learned during training. These patterns, or “syntactic templates,” can become incorrectly linked to certain topics. As a result, a model may generate confident-sounding answers simply because it recognizes a familiar sentence structure, not because it actually understands the question.
The team found that even state-of-the-art LLMs exhibit this behavior, which can lead to unexpected failures across tasks such as handling customer queries, summarising clinical notes, or preparing financial reports. The flaw also carries safety implications: attackers could potentially exploit these syntactic associations to bypass built-in safeguards and trigger harmful responses.
To study the problem, researchers created controlled experiments in which each domain used only one syntactic pattern during training. When tested with sentences that preserved the syntax but replaced meaningful words with synonyms, antonyms, or even random terms, the models still often returned the “correct” domain-specific answer — even when the question itself was gibberish.
“This is a byproduct of how we train models, but models are now used in practice in safety-critical domains far beyond the tasks that created these syntactic failure modes,” said Marzyeh Ghassemi, associate professor at MIT’s Department of Electrical Engineering and Computer Science and the senior author of the study. “If you’re not familiar with model training as an end-user, this is likely to be unexpected.”
The researchers have also developed a new benchmarking procedure to measure the degree to which a model relies on such faulty correlations. They say this could help developers detect and mitigate the issue before deploying models in high-stakes settings.
The study was co-authored by Chantal Shaib of Northeastern University, Vinith Suriyakumar of MIT, Meta researcher Levent Sagun, and Northeastern professor Byron Wallace. The findings will be presented at the Conference on Neural Information Processing Systems (NeurIPS).
Author
