Reasoning skills of large language models are often overestimated

Large language models (LLMs) like GPT-4 and Claude excel in familiar tasks but struggle in novel scenarios, revealing a reliance on memorization over reasoning skills, according to recent MIT CSAIL research. The study compared default tasks with counterfactual scenarios, showing that LLMs perform well in common tasks but experience significant performance drops in unfamiliar situations. Tasks like arithmetic, chess, and logical reasoning were used to test the models' abilities, highlighting limitations in generalization to new scenarios. The research suggests that while LLMs may perform strongly in known tasks, their adaptability to diverse situations is limited. The study emphasizes the importance of enhancing LLMs' adaptability and understanding their decision-making processes. Despite the insights gained, the study acknowledges the need for more diverse testing environments to uncover additional weaknesses and improve the interpretability of these models. The research was presented at the North American Chapter of the Association for Computational Linguistics (NAACL) and was supported by various institutions including the MIT–IBM Watson AI Lab and the National Science Foundation.