Empowering Developers with Local Reasoning Model Training Using Only 5GB VRAM
In recent weeks, our community has shown tremendous support for our latest release, and today, we’re thrilled to share a groundbreaking advancement that brings powerful reasoning model training within reach for a broader audience. Specifically, you can now train your own reasoning models locally with as little as 5GB of VRAM — a significant reduction from previous requirements.
Introducing the Efficient GRPO Algorithm
This leap forward is made possible by our newly developed Efficient Guided Reinforcement Policy Optimization (GRPO) algorithm. This innovative approach enables:
- Tenfold increase in context length, supporting longer sequences for more complex reasoning tasks.
- 90% reduction in VRAM consumption compared to traditional GRPO LoRA/QLoRA implementations, even those leveraging advanced techniques like Flash Attention 2 (FA2).
For example, training the Qwen2.5 (1.5B parameters) model now requires only around 5GB VRAM, down from 7GB in earlier releases, making it much more accessible.
Practical Implications for Large-Scale Model Training
Consider the resource demands of training Llama 3.1 (8B parameters) at a 20,000 token context length. Using conventional setups with TRL + FA2 would demand approximately 510.8GB of VRAM. Thanks to our optimized algorithms, this requirement drops to a mere 54.3GB, opening doors for more researchers and developers to experiment locally.
Advanced Memory Optimization with Gradient Checkpointing
Our custom gradient checkpointing technique further reduces memory needs by asynchronously offloading intermediate activations to system RAM, without sacrificing much in speed (only about 1% slower). This approach effectively saves up to 372GB of VRAM, especially vital when performing multiple generations (e.g., eight in this case). Additionally, gradient accumulation offers an avenue for even more memory savings.
Get Started Today
To experience these advancements firsthand, try our free GRPO notebook featuring 10x longer context support:
For a deeper dive into the algorithm, the underlying mathematics, encountered challenges, and solutions, visit our detailed blog post:
VRAM Usage Breakdown
| Metric | Unsloth Approach | TRL + FA2 Approach |
|—|—|—|
| Training Memory Cost | 42GB | 414GB |
| GRPO Memory Cost | 9.8GB | 78.3GB |
| Inference Cost | 0GB | 16GB |
| KV Cache for 20K Context | 2.5GB | 2.5GB |
| Total Memory Usage | 54.3GB (90% less) | 510.8GB |
Additional Enhancements
- Full Reward Function Logging: We now provide comprehensive logs for all reward functions, enhancing transparency and debugging capabilities.
- 4-bit Dynamic Quantization: Perform inference with our dynamic 4-bit quantization directly in vLLM, further optimizing resource use.
- Extensive Guides and Documentation: Our updated GRPO and reward function guide offers detailed insights into setup, tuning, and best practices.
We sincerely appreciate the continued support from our community. Your enthusiasm inspires us to push the boundaries of what’s possible. Stay tuned for our upcoming major release, which promises even more powerful features and improvements.
Thank you for being part of this journey!
— The Unsloth Team
