TheBloke/Puma-3b-GGML

Chat & support: TheBloke's Discord server

Want to contribute? TheBloke's Patreon page

TheBloke's LLM work is generously supported by a grant from andreessen horowitz (a16z)

---

Puma 3B - GGML

• Model creator: Bohan Du
• Original model: Puma 3B

Description

This repo contains GGML format model files for Bohan Du's Puma 3B.

GGML files are for CPU + GPU inference using llama.cpp and libraries and UIs which support this format, such as:

• text-generation-webui, the most popular web UI. Supports NVidia CUDA GPU acceleration.
• KoboldCpp, a powerful GGML web UI with GPU acceleration on all platforms (CUDA and OpenCL). Especially good for story telling.
• LM Studio, a fully featured local GUI with GPU acceleration on both Windows (NVidia and AMD), and macOS.
• LoLLMS Web UI, a great web UI with CUDA GPU acceleration via the c_transformers backend.
• ctransformers, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
• llama-cpp-python, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.

Repositories available

• GPTQ models for GPU inference, with multiple quantisation parameter options.
• 2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference
• Bohan Du's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

Prompt template: Human-Response

### HUMAN:
{prompt}

### RESPONSE:

Compatibility

These quantised GGML files are compatible with llama.cpp as of June 6th, commit 2d43387.

They should also be compatible with all UIs, libraries and utilities which use GGML.

Explanation of the new k-quant methods

Click to see details

The new methods available are:

• GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
• GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
• GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
• GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
• GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
• GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.

Refer to the Provided Files table below to see what files use which methods, and how.

Provided files

Name	Quant method	Bits	Size	Max RAM required	Use case
puma-3b.ggmlv3.q4_0.bin	q4_0	4	1.93 GB	4.43 GB	Original quant method, 4-bit.
puma-3b.ggmlv3.q4_1.bin	q4_1	4	2.14 GB	4.64 GB	Original quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
puma-3b.ggmlv3.q5_0.bin	q5_0	5	2.36 GB	4.86 GB	Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference.
puma-3b.ggmlv3.q5_1.bin	q5_1	5	2.57 GB	5.07 GB	Original quant method, 5-bit. Even higher accuracy, resource usage and slower inference.
puma-3b.ggmlv3.q8_0.bin	q8_0	8	3.64 GB	6.14 GB	Original quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users.

Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.

How to run in llama.cpp

I use the following command line; adjust for your tastes and needs:

./main -t 10 -ngl 32 -m puma-3b.ggmlv3.q4_K_M.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"

Change -t 10 to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use -t 8.

Change -ngl 32 to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.

Change -c 2048 to the desired sequence length for this model. For example, -c 4096 for a Llama 2 model. For models that use RoPE, add --rope-freq-base 10000 --rope-freq-scale 0.5 for doubled context, or --rope-freq-base 10000 --rope-freq-scale 0.25 for 4x context.

If you want to have a chat-style conversation, replace the -p <PROMPT> argument with -i -ins

For other parameters and how to use them, please refer to the llama.cpp documentation

How to run in text-generation-webui

Further instructions here: text-generation-webui/docs/llama.cpp.md.

Discord

For further support, and discussions on these models and AI in general, join us at:

TheBloke AI's Discord server

Thanks, and how to contribute.

Thanks to the chirper.ai team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.

• Patreon: https://patreon.com/TheBlokeAI
• Ko-Fi: https://ko-fi.com/TheBlokeAI

Special thanks to: Aemon Algiz.

Patreon special mentions: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter

Thank you to all my generous patrons and donaters!

And thank you again to a16z for their generous grant.

Original model card: Bohan Du's Puma 3B

This is OpenLLaMA 3B V2 finetuned on ShareGPT Hyperfiltered for 1 epochs.

Prompt template:

### HUMAN:
{prompt}

### RESPONSE:
<leave a newline for the model to answer>

q4_1 GGML quant available here.

Note: Don't expect this model to be good, I was just starting out to finetune. So don't roast me please!

Benchmarks:

arc_challenge	acc 0.3660409556313993 acc_stderr 0.014077223108470142 acc_norm 0.4121160409556314 acc_norm_stderr 0.014383915302225403
hellaswag	acc 0.5401314479187412 acc_stderr 0.004973683026202171 acc_norm 0.7177853017327226 acc_norm_stderr 0.004491574539441884
hendrycksTest-abstract_algebra	acc 0.21 acc_stderr 0.04093601807403326 acc_norm 0.21 acc_norm_stderr 0.04093601807403326
hendrycksTest-anatomy	acc 0.25925925925925924 acc_stderr 0.03785714465066653 acc_norm 0.25925925925925924 acc_norm_stderr 0.03785714465066653
hendrycksTest-astronomy	acc 0.28289473684210525 acc_stderr 0.03665349695640767 acc_norm 0.28289473684210525 acc_norm_stderr 0.03665349695640767
hendrycksTest-business_ethics	acc 0.34 acc_stderr 0.04760952285695236 acc_norm 0.34 acc_norm_stderr 0.04760952285695236
hendrycksTest-clinical_knowledge	acc 0.2528301886792453 acc_stderr 0.02674989977124123 acc_norm 0.2528301886792453 acc_norm_stderr 0.02674989977124123
hendrycksTest-college_biology	acc 0.24305555555555555 acc_stderr 0.03586879280080339 acc_norm 0.24305555555555555 acc_norm_stderr 0.03586879280080339
hendrycksTest-college_chemistry	acc 0.19 acc_stderr 0.03942772444036623 acc_norm 0.19 acc_norm_stderr 0.03942772444036623
hendrycksTest-college_computer_science	acc 0.21 acc_stderr 0.040936018074033256 acc_norm 0.21 acc_norm_stderr 0.040936018074033256
hendrycksTest-college_mathematics	acc 0.36 acc_stderr 0.04824181513244218 acc_norm 0.36 acc_norm_stderr 0.04824181513244218
hendrycksTest-college_medicine	acc 0.20809248554913296 acc_stderr 0.030952890217749877 acc_norm 0.20809248554913296 acc_norm_stderr 0.030952890217749877
hendrycksTest-college_physics	acc 0.2647058823529412 acc_stderr 0.04389869956808778 acc_norm 0.2647058823529412 acc_norm_stderr 0.04389869956808778
hendrycksTest-computer_security	acc 0.33 acc_stderr 0.047258156262526045 acc_norm 0.33 acc_norm_stderr 0.047258156262526045
hendrycksTest-conceptual_physics	acc 0.33617021276595743 acc_stderr 0.030881618520676942 acc_norm 0.33617021276595743 acc_norm_stderr 0.030881618520676942
hendrycksTest-econometrics	acc 0.24561403508771928 acc_stderr 0.04049339297748144 acc_norm 0.24561403508771928 acc_norm_stderr 0.04049339297748144
hendrycksTest-electrical_engineering	acc 0.2689655172413793 acc_stderr 0.03695183311650232 acc_norm 0.2689655172413793 acc_norm_stderr 0.03695183311650232
hendrycksTest-elementary_mathematics	acc 0.2724867724867725 acc_stderr 0.022930973071633366 acc_norm 0.2724867724867725 acc_norm_stderr 0.022930973071633366
hendrycksTest-formal_logic	acc 0.21428571428571427 acc_stderr 0.03670066451047181 acc_norm 0.21428571428571427 acc_norm_stderr 0.03670066451047181
hendrycksTest-global_facts	acc 0.32 acc_stderr 0.046882617226215034 acc_norm 0.32 acc_norm_stderr 0.046882617226215034
hendrycksTest-high_school_biology	acc 0.24193548387096775 acc_stderr 0.024362599693031093 acc_norm 0.24193548387096775 acc_norm_stderr 0.024362599693031093
hendrycksTest-high_school_chemistry	acc 0.26108374384236455 acc_stderr 0.030903796952114485 acc_norm 0.26108374384236455 acc_norm_stderr 0.030903796952114485
hendrycksTest-high_school_computer_science	acc 0.27 acc_stderr 0.04461960433384739 acc_norm 0.27 acc_norm_stderr 0.04461960433384739
hendrycksTest-high_school_european_history	acc 0.30303030303030304 acc_stderr 0.035886248000917075 acc_norm 0.30303030303030304 acc_norm_stderr 0.035886248000917075
hendrycksTest-high_school_geography	acc 0.24242424242424243 acc_stderr 0.030532892233932026 acc_norm 0.24242424242424243 acc_norm_stderr 0.030532892233932026
hendrycksTest-high_school_government_and_politics	acc 0.24352331606217617 acc_stderr 0.030975436386845426 acc_norm 0.24352331606217617 acc_norm_stderr 0.030975436386845426
hendrycksTest-high_school_macroeconomics	acc 0.2564102564102564 acc_stderr 0.022139081103971545 acc_norm 0.2564102564102564 acc_norm_stderr 0.022139081103971545
hendrycksTest-high_school_mathematics	acc 0.2518518518518518 acc_stderr 0.026466117538959905 acc_norm 0.2518518518518518 acc_norm_stderr 0.026466117538959905
hendrycksTest-high_school_microeconomics	acc 0.24789915966386555 acc_stderr 0.028047967224176892 acc_norm 0.24789915966386555 acc_norm_stderr 0.028047967224176892
hendrycksTest-high_school_physics	acc 0.31125827814569534 acc_stderr 0.037804458505267334 acc_norm 0.31125827814569534 acc_norm_stderr 0.037804458505267334
hendrycksTest-high_school_psychology	acc 0.26422018348623855 acc_stderr 0.01890416417151019 acc_norm 0.26422018348623855 acc_norm_stderr 0.01890416417151019
hendrycksTest-high_school_statistics	acc 0.2175925925925926 acc_stderr 0.028139689444859672 acc_norm 0.2175925925925926 acc_norm_stderr 0.028139689444859672
hendrycksTest-high_school_us_history	acc 0.24019607843137256 acc_stderr 0.02998373305591362 acc_norm 0.24019607843137256 acc_norm_stderr 0.02998373305591362
hendrycksTest-high_school_world_history	acc 0.28270042194092826 acc_stderr 0.029312814153955924 acc_norm 0.28270042194092826 acc_norm_stderr 0.029312814153955924
hendrycksTest-human_aging	acc 0.3901345291479821 acc_stderr 0.03273766725459156 acc_norm 0.3901345291479821 acc_norm_stderr 0.03273766725459156
hendrycksTest-human_sexuality	acc 0.1984732824427481 acc_stderr 0.03498149385462473 acc_norm 0.1984732824427481 acc_norm_stderr 0.03498149385462473
hendrycksTest-international_law	acc 0.30578512396694213 acc_stderr 0.04205953933884124 acc_norm 0.30578512396694213 acc_norm_stderr 0.04205953933884124
hendrycksTest-jurisprudence	acc 0.24074074074074073 acc_stderr 0.0413311944024384 acc_norm 0.24074074074074073 acc_norm_stderr 0.0413311944024384
hendrycksTest-logical_fallacies	acc 0.22085889570552147 acc_stderr 0.03259177392742177 acc_norm 0.22085889570552147 acc_norm_stderr 0.03259177392742177
hendrycksTest-machine_learning	acc 0.24107142857142858 acc_stderr 0.04059867246952685 acc_norm 0.24107142857142858 acc_norm_stderr 0.04059867246952685
hendrycksTest-management	acc 0.2912621359223301 acc_stderr 0.04498676320572922 acc_norm 0.2912621359223301 acc_norm_stderr 0.04498676320572922
hendrycksTest-marketing	acc 0.28205128205128205 acc_stderr 0.029480360549541194 acc_norm 0.28205128205128205 acc_norm_stderr 0.029480360549541194
hendrycksTest-medical_genetics	acc 0.31 acc_stderr 0.04648231987117316 acc_norm 0.31 acc_norm_stderr 0.04648231987117316
hendrycksTest-miscellaneous	acc 0.30140485312899107 acc_stderr 0.016409091097268787 acc_norm 0.30140485312899107 acc_norm_stderr 0.016409091097268787
hendrycksTest-moral_disputes	acc 0.30346820809248554 acc_stderr 0.024752411960917202 acc_norm 0.30346820809248554 acc_norm_stderr 0.024752411960917202
hendrycksTest-moral_scenarios	acc 0.24692737430167597 acc_stderr 0.014422292204808835 acc_norm 0.24692737430167597 acc_norm_stderr 0.014422292204808835
hendrycksTest-nutrition	acc 0.28431372549019607 acc_stderr 0.025829163272757475 acc_norm 0.28431372549019607 acc_norm_stderr 0.025829163272757475
hendrycksTest-philosophy	acc 0.29260450160771706 acc_stderr 0.02583989833487798 acc_norm 0.29260450160771706 acc_norm_stderr 0.02583989833487798
hendrycksTest-prehistory	acc 0.2993827160493827 acc_stderr 0.02548311560119547 acc_norm 0.2993827160493827 acc_norm_stderr 0.02548311560119547
hendrycksTest-professional_accounting	acc 0.2624113475177305 acc_stderr 0.02624492034984301 acc_norm 0.2624113475177305 acc_norm_stderr 0.02624492034984301
hendrycksTest-professional_law	acc 0.24641460234680573 acc_stderr 0.011005971399927242 acc_norm 0.24641460234680573 acc_norm_stderr 0.011005971399927242
hendrycksTest-professional_medicine	acc 0.21691176470588236 acc_stderr 0.02503584522771125 acc_norm 0.21691176470588236 acc_norm_stderr 0.02503584522771125
hendrycksTest-professional_psychology	acc 0.27941176470588236 acc_stderr 0.018152871051538812 acc_norm 0.27941176470588236 acc_norm_stderr 0.018152871051538812
hendrycksTest-public_relations	acc 0.2636363636363636 acc_stderr 0.04220224692971987 acc_norm 0.2636363636363636 acc_norm_stderr 0.04220224692971987
hendrycksTest-security_studies	acc 0.37551020408163266 acc_stderr 0.031001209039894836 acc_norm 0.37551020408163266 acc_norm_stderr 0.031001209039894836
hendrycksTest-sociology	acc 0.26865671641791045 acc_stderr 0.03134328358208954 acc_norm 0.26865671641791045 acc_norm_stderr 0.03134328358208954
hendrycksTest-us_foreign_policy	acc 0.35 acc_stderr 0.0479372485441102 acc_norm 0.35 acc_norm_stderr 0.0479372485441102
hendrycksTest-virology	acc 0.3253012048192771 acc_stderr 0.03647168523683227 acc_norm 0.3253012048192771 acc_norm_stderr 0.03647168523683227
hendrycksTest-world_religions	acc 0.30409356725146197 acc_stderr 0.035282112582452334 acc_norm 0.30409356725146197 acc_norm_stderr 0.035282112582452334
truthfulqa_mc	mc1 0.2460220318237454 mc1_stderr 0.015077219200662597 mc2 0.3821583918021301 mc2_stderr 0.013892826466923648