Overview
PowerInfer developed and released the SmallThinker-3B-preview, a fine-tuned version of the Qwen2.5-3B-Instruct model. SmallThinker is optimized for efficient deployment on resource-constrained devices while maintaining high performance in reasoning, coding, and general text generation tasks. It outperforms its base model on key benchmarks, including AIME24, AMC23, and GAOKAO2024, making it a robust tool for both edge deployment and as a draft model for larger systems like QwQ-32B-Preview.
SmallThinker was fine-tuned in two phases using high-quality datasets, including PowerInfer/QWQ-LONGCOT-500K and PowerInfer/LONGCOT-Refine-500K. Its small size allows for up to 70% faster inference speeds compared to larger models, making it ideal for applications requiring quick responses and efficient computation.
Variants
| No | Variant | Cortex CLI command |
|---|---|---|
| 1 | gguf | cortex run small-thinker |
Use it with Jan (UI)
- Install Jan using Quickstart
- Use in Jan model Hub:
cortexso/small-thinker
Use it with Cortex (CLI)
- Install Cortex using Quickstart
- Run the model with command:
cortex run small-thinker