Dola Decoding by Contrasting Layers Improves Factuality in Large Language Models
A decoding strategy named DoLa reduces hallucinations in large language models without external knowledge. It contrasts logits from different layers to enhance truthfulness, improving factual generation by 12-17% in tasks like TruthfulQA.
Read original article
The paper titled "DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models" introduces a decoding strategy to reduce hallucinations in large language models (LLMs) without the need for external knowledge or additional fine-tuning. The proposed approach, called Decoding by Contrasting Layers (DoLa), contrasts the differences in logits from later layers with earlier layers to surface factual knowledge localized within specific transformer layers. This method enhances truthfulness in LLMs by reducing the generation of incorrect facts. The DoLa approach consistently improves truthfulness in various tasks, such as TruthfulQA, by 12-17% absolute points, showcasing its potential to enhance the reliability of LLMs in generating truthful information. The paper was presented at the ICLR 2024 main conference and provides a source code available for further exploration.
Related
Researchers describe how to tell if ChatGPT is confabulating
Researchers at the University of Oxford devised a method to detect confabulation in large language models like ChatGPT. By assessing semantic equivalence, they aim to reduce false answers and enhance model accuracy.
Detecting hallucinations in large language models using semantic entropy
Researchers devised a method to detect hallucinations in large language models like ChatGPT and Gemini by measuring semantic entropy. This approach enhances accuracy by filtering unreliable answers, improving model performance significantly.
Large Language Models are not a search engine
Large Language Models (LLMs) from Google and Meta generate algorithmic content, causing nonsensical "hallucinations." Companies struggle to manage errors post-generation due to factors like training data and temperature settings. LLMs aim to improve user interactions but raise skepticism about delivering factual information.
Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs
The study presents a method to boost Large Language Models' retrieval and reasoning abilities for long-context inputs by fine-tuning on a synthetic dataset. Results show significant improvements in information retrieval and reasoning skills.
Reasoning in Large Language Models: A Geometric Perspective
The paper delves into how large language models reason geometrically, linking self-attention graph density to expressive power. Higher intrinsic dimensions enhance LLMs' capacity, supported by theoretical, toy examples, and empirical evidence.
3 commentsThis is surprising
Related
Researchers describe how to tell if ChatGPT is confabulating
Researchers at the University of Oxford devised a method to detect confabulation in large language models like ChatGPT. By assessing semantic equivalence, they aim to reduce false answers and enhance model accuracy.
Detecting hallucinations in large language models using semantic entropy
Researchers devised a method to detect hallucinations in large language models like ChatGPT and Gemini by measuring semantic entropy. This approach enhances accuracy by filtering unreliable answers, improving model performance significantly.
Large Language Models are not a search engine
Large Language Models (LLMs) from Google and Meta generate algorithmic content, causing nonsensical "hallucinations." Companies struggle to manage errors post-generation due to factors like training data and temperature settings. LLMs aim to improve user interactions but raise skepticism about delivering factual information.
Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs
The study presents a method to boost Large Language Models' retrieval and reasoning abilities for long-context inputs by fine-tuning on a synthetic dataset. Results show significant improvements in information retrieval and reasoning skills.
Reasoning in Large Language Models: A Geometric Perspective
The paper delves into how large language models reason geometrically, linking self-attention graph density to expressive power. Higher intrinsic dimensions enhance LLMs' capacity, supported by theoretical, toy examples, and empirical evidence.