Development of method for self-correction of large language models via reinforcement learning

Abstract

This work presents a reinforcement learning-based approach to the self-correction of large language models (LLMs). The motivation stems from the instability in output quality of current LLMs, their tendency to produce factual or logical errors, and the lack of built-in mechanisms for verifying and improving their own responses. We formalize self-correction as an episodic Markov Decision Process (MDP), where the model generates an initial response and a subsequent correction attempt, with a binary reward signal based on the improvement in quality. The study addresses major challenges in this setting, including distributional shift, behavioral collapse, and reward hacking. We survey existing approaches, such as test-time reasoning, chain-ofthought prompting, fine-tuning, and reinforcement learningbased strategies. Our proposed method employs the Advantage Actor-Critic (A2C) algorithm, selected for its balance between implementation simplicity and optimization efficiency. We describe a two-stage training architecture designed to foster stable selfcorrection capabilities. Experimental results on mathematical problem-solving tasks demonstrate notable improvements in response quality. These findings highlight the practical relevance of the proposed method, showing enhanced reliability and robustness in the outputs of large language models.