New GGMLv3 format for breaking llama.cpp change May 19th commit 2d5db48

README.md

@@ -26,29 +26,28 @@ GGML files are for CPU inference using [llama.cpp](https://github.com/ggerganov/
 * [4bit and 5bit GGML models for CPU inference](https://huggingface.co/TheBloke/h2ogpt-oasst1-512-30B-GGML).
 * [float16 HF format unquantised model for GPU inference and further conversions](https://huggingface.co/TheBloke/h2ogpt-oasst1-512-30B-HF)
 
-## THESE FILES REQUIRE LATEST LLAMA.CPP (May 12th 2023 - commit b9fd7ee)!
+## THE FILES IN MAIN BRANCH REQUIRES LATEST LLAMA.CPP (May 19th 2023 - commit 2d5db48)!
 
-llama.cpp recently made a breaking change to its quantisation methods.
+llama.cpp recently made another breaking change to its quantisation methods - https://github.com/ggerganov/llama.cpp/pull/1508
 
-I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 12th or later (commit `b9fd7ee` or later) to use them.
-
-If you are currently unable to update llama.cpp, eg because you use a UI which hasn't updated yet, you can find GGML files for the previous version of llama.cpp in the `previous_llama` branch.
+I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 19th or later (commit `2d5db48` or later) to use them.
 
+For files compatible with the previous version of llama.cpp, please see branch `previous_llama_ggmlv2`.
 ## Provided files
 | Name | Quant method | Bits | Size | RAM required | Use case |
 | ---- | ---- | ---- | ---- | ---- | ----- |
-`h2ogptq-oasst1-512-30B.ggml.q4_0.bin` | q4_0 | 4bit | 20.3GB | 25GB | 4-bit. |
-`h2ogptq-oasst1-512-30B.ggml.q4_1.bin` | q4_1 | 4bit | 24.4GB | 26GB | 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
-`h2ogptq-oasst1-512-30B.ggml.q5_0.bin` | q5_0 | 5bit | 22.4GB | 25GB | 5-bit. Higher accuracy, higher resource usage and slower inference. |
-`h2ogptq-oasst1-512-30B.ggml.q5_1.bin` | q5_1 | 5bit | 24.4GB | 26GB | 5-bit. Even higher accuracy, and higher resource usage and slower inference.|
-`h2ogptq-oasst1-512-30B.ggml.q8_0.bin` | q8_0 | 8bit | 36.6GB | 39GB | 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. |
+`h2ogptq-oasst1-512-30B.ggmlv3.q4_0.bin` | q4_0 | 4bit | 20.3GB | 25GB | 4-bit. |
+`h2ogptq-oasst1-512-30B.ggmlv3.q4_1.bin` | q4_1 | 4bit | 24.4GB | 26GB | 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
+`h2ogptq-oasst1-512-30B.ggmlv3.q5_0.bin` | q5_0 | 5bit | 22.4GB | 25GB | 5-bit. Higher accuracy, higher resource usage and slower inference. |
+`h2ogptq-oasst1-512-30B.ggmlv3.q5_1.bin` | q5_1 | 5bit | 24.4GB | 26GB | 5-bit. Even higher accuracy, and higher resource usage and slower inference.|
+`h2ogptq-oasst1-512-30B.ggmlv3.q8_0.bin` | q8_0 | 8bit | 36.6GB | 39GB | 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. |
 
 ## How to run in `llama.cpp`
 
 I use the following command line; adjust for your tastes and needs:
 
 ```
-./main -t 8 -m h2ogptq-oasst1-512-30B.ggml.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request.
+./main -t 8 -m h2ogptq-oasst1-512-30B.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request.
 ### Instruction:
 Write a story about llamas
 ### Response:"
@@ -63,6 +62,8 @@ GGML models can be loaded into text-generation-webui by installing the llama.cpp
 
 Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
 
+Note: at this time text-generation-webui may not support the new May 19th llama.cpp quantisation methods for q4_0, q4_1 and q8_0 files.
+
 # Original h2oGPT Model Card
 ## Summary
 
@@ -284,4 +285,4 @@ Please read this disclaimer carefully before using the large language model prov
 - Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
 - Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.
 
-By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.
+By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.