hongyuw
Abstract
Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.
Community
Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.
are you going to release the model trained on 2T tokens?
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms (2024)
- GWQ: Gradient-Aware Weight Quantization for Large Language Models (2024)
- 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs (2024)
- FlatQuant: Flatness Matters for LLM Quantization (2024)
- PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
@hongyuw just a small thing, would be cool to have the same x axis ranges in the plots in figure 2 and 3, same for the y-axes (ppl) in figure 4 and 5.
besides that, really cool stuff! do you have measurements on inference speed (tokens/s throughput) and memory footprint during inference in comparison to the original model and the 1.58bit model as well to compare? and will you make your code for it available somewhere?
@hongyuw just a small thing, would be cool to have the same x axis ranges in the plots in figure 2 and 3, same for the y-axes (ppl) in figure 4 and 5.
besides that, really cool stuff! do you have measurements on inference speed (tokens/s throughput) and memory footprint during inference in comparison to the original model and the 1.58bit model as well to compare? and will you make your code for it available somewhere?
Thanks for your suggestions.
We are working on the inference kernel of BitNet a4.8 on GPU/CPU devices, and will release it in the bitnet.cpp. Please stay tuned!
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper