qwp4w3hyb
/

Mistral-Nemo-Instruct-2407-iMat-GGUF

Text Generation

Inference Endpoints

Model card Files Files and versions Community

qwp4w3hyb commited on Jul 24, 2024

Commit

4a4f424

·

verified ·

1 Parent(s): 2db829e

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -16,7 +16,7 @@ base_model: mistralai/Mistral-Nemo-Instruct-2407
 - needs [#8604](https://github.com/ggerganov/llama.cpp/pull/8604) & latest master with tekken tokenizer fixes applied
 - quants done with an importance matrix for improved quantization loss
-- Requantized ggufs & imatrix from hf bf16
 - Wide coverage of different gguf quant types from Q\_8\_0 down to IQ1\_S
   - experimental custom quant types
     - `_L` with `--output-tensor-type f16 --token-embedding-type f16` (same as bartowski's)

 - needs [#8604](https://github.com/ggerganov/llama.cpp/pull/8604) & latest master with tekken tokenizer fixes applied
 - quants done with an importance matrix for improved quantization loss
+- Quantized ggufs & imatrix from hf bf16, through bf16. `safetensors bf16 -> gguf bf16 -> quant` for optimal quant loss.
 - Wide coverage of different gguf quant types from Q\_8\_0 down to IQ1\_S
   - experimental custom quant types
     - `_L` with `--output-tensor-type f16 --token-embedding-type f16` (same as bartowski's)