DeepSeek R1 Paper Innovation Summary

[Innovation Points]
1. Pure reinforcement learning-driven reasoning improvement:

DeepSeek-R1-Zero is the first model trained entirely through large-scale reinforcement learning (RL) without supervised fine-tuning (SFT). Its innovation lies in validating that pure RL can incentivize large language models (LLMs) to autonomously develop complex reasoning behaviors (such as self-verification and long chain-of-thought generation), breaking the traditional paradigm that relies on SFT.

During training, the model naturally exhibits behaviors such as "reflection" and "multilingual mixed problem-solving," and significantly improves performance through RL (e.g., AIME 2024 Pass@1 increased from 15.6% to 71.0%).

2. Cold start and multi-stage training pipeline:

DeepSeek-R1 introduces a small amount of manually designed cold-start data (such as readable CoT format) and multi-stage training (cold-start SFT → RL → re-sampling SFT → full-scenario RL), solving R1-Zero's poor readability and language mixing issues, while further improving performance (AIME 2024 Pass@1 reaches 79.8%, comparable to OpenAI-o1-1217).

3. Distilling reasoning capabilities into smaller models:

By directly distilling the reasoning data generated by DeepSeek-R1 into smaller models in the Qwen and Llama series (such as 7B and 32B), the reasoning capabilities of small models are significantly improved. For example, the distilled Qwen-32B achieves a Pass@1 of 72.6% on AIME 2024, far exceeding the baseline of the same scale (QwQ-32B-Preview only 50.0%).