Prover-Verifier Games improve legibility of LLM outputs

The paper titled "Prover-Verifier Games improve legibility of LLM outputs" by Jan Hendrik Kirchner and colleagues explores enhancing the legibility of outputs from Large Language Models (LLMs). The authors argue that clear and easily verifiable reasoning can boost confidence in LLM outputs, particularly in solving grade-school math problems. They note that focusing solely on answer correctness can diminish legibility. To address this, they propose a training algorithm inspired by Prover-Verifier Games, which involves training small verifiers to assess solution correctness, alongside "helpful" provers that generate correct solutions and "sneaky" provers that produce incorrect ones to challenge the verifiers. The study finds that the accuracy of helpful provers and the robustness of verifiers against adversarial attacks improve with training. Additionally, the training enhances human accuracy when verifying the solutions of helpful provers, while accuracy declines with sneaky provers. This suggests that training LLMs for checkability through small verifiers is a viable method for improving output legibility, which could facilitate better alignment of advanced models with human understanding. The findings indicate that legibility training could be a practical approach to enhance the interpretability of LLM outputs, ultimately benefiting users who rely on these models for accurate information.