Mid-range models exl2 quants
Collection
Exl2 quants of mid-range (20-40B) LLM models. Usually around 4-5 BPW
•
8 items
•
Updated
Configuration Parsing
Warning:
In config.json: "quantization_config.bits" must be an integer
Mistral-Small-Instruct-2409 - EXL2 6.8 rpcal_mk2
This is a 8bpw EXL2 quant of mistralai/Mistral-Small-Instruct-2409
This quant was made using exllamav2-0.2.2 with Fullmoon-light dataset for RP.
I tested this quant shortly in some random RPs (including ones over 8k and 16k context) and it seems to work fine.
Prompt Templates
Uses Mistral format.
For more details see original model.
- Downloads last month
- 5
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.