---
license: mit
license_name: mit
license_link: LICENSE
library_name: transformers
tags:
    - fp8
    - vllm
language:
    - en
    - de
    - fr
    - it
    - pt
    - hi
    - es
    - th
pipeline_tag: text-generation
base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
---

# DeepSeek-R1-Distill-Qwen-14B-FP8

FP8-quantized version of [DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B), optimized for inference with vLLM. The quantization reduces the model's memory footprint by approximately 50%.

## Model Overview

-   **Base Model**: DeepSeek-R1-Distill-Qwen-14B
-   **Quantization**: FP8 (weights and activations)
-   **Memory Reduction**: ~50% (from 16-bit to 8-bit)
-   **License**: MIT License (following original model's license)

## Compression Details

Compressed using [LLM Compressor](https://github.com/vllm-project/llm-compressor) with:

-   512 calibration samples from UltraChat
-   Symmetric per-tensor quantization
-   Applied to linear operators within transformer blocks

The compression script is available in `compress.py`.

## Requirements

-   vLLM
-   transformers
-   torch
-   accelerate

## Note

This is an experimental compression of the model. Performance metrics and optimal usage parameters have not been thoroughly tested yet.