Hugging Face's logo

mo137
/
Amethyst-13B-Mistral-6bpw-exl2

Text Generation
Transformers
Safetensors
llama
exllamav2
exl2
Text Generation
Not-For-All-Audiences
nsfw
Transformers
text-generation-inference
Inference Endpoints
Model card Files Community
Edit model card

Amethyst 13B Mistral - EXL2 - 6 bpw

Description

  • 6 bits per weight.

I converted the model using the convert.py script from the exllamav2 repo:
https://github.com/turboderp/exllamav2
Its documentation:
https://github.com/turboderp/exllamav2/blob/master/doc/convert.md

I used the WikiText-2-v1 dataset for calibration:
https://huggingface.co/datasets/wikitext/blob/refs%2Fconvert%2Fparquet/wikitext-2-v1/test/0000.parquet

Downloads last month
7
Inference Examples
Text Generation
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.