Hugging Face's logo

second-state
/
openchat-3.5-1210-GGUF

Text Generation
Transformers
GGUF
mistral
openchat
conversational
Model card Files Community
Xin Liu commited on
Commit
c8ccff4
·
1 Parent(s): 14f5363

Update

Browse files

Signed-off-by: Xin Liu <[email protected]>

Files changed (2) hide show
  1. README.md +61 -6
  2. openchat-3.5-1210.Q5_K_M.gguf +0 -3
README.md CHANGED
@@ -1,15 +1,70 @@
1
- # OpenChat-3.5-1210
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
 
3
  ## Original Model
4
 
5
  [openchat/openchat-3.5-1210](https://huggingface.co/openchat/openchat-3.5-1210)
6
 
7
- ## Prompt Template
 
 
 
 
 
 
 
 
8
 
9
- - Prompt type: `openchat`
 
 
10
 
11
- - Prompt string
12
 
13
- ```text
14
- GPT4 User: {prompt}<|end_of_turn|>GPT4 Assistant:
15
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: openchat/openchat-3.5-1210
3
+ inference: false
4
+ library_name: transformers
5
+ license: apache-2.0
6
+ model_creator: OpenChat
7
+ model_name: Openchat 3.5 1210
8
+ model_type: mistral
9
+ pipeline_tag: text-generation
10
+ quantized_by: Second State Inc.
11
+ tags:
12
+ - openchat
13
+ ---
14
+
15
+ <!-- header start -->
16
+ <!-- 200823 -->
17
+ <div style="width: auto; margin-left: auto; margin-right: auto">
18
+ <img src="https://github.com/second-state/LlamaEdge/raw/dev/assets/logo.svg" style="width: 100%; min-width: 400px; display: block; margin: auto;">
19
+ </div>
20
+ <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
21
+ <!-- header end -->
22
+
23
+ # OpenChat-3.5-0106-GGUF
24
 
25
  ## Original Model
26
 
27
  [openchat/openchat-3.5-1210](https://huggingface.co/openchat/openchat-3.5-1210)
28
 
29
+ ## Run with LlamaEdge
30
+
31
+ - LlamaEdge version: [v0.2.4](https://github.com/second-state/LlamaEdge/releases/tag/0.2.4)
32
+
33
+ - Prompt template
34
+
35
+ - Prompt type: `openchat`
36
+
37
+ - Prompt string
38
 
39
+ ```text
40
+ GPT4 User: {prompt}<|end_of_turn|>GPT4 Assistant:
41
+ ```
42
 
43
+ - Run as LlamaEdge service
44
 
45
+ ```bash
46
+ wasmedge --dir .:. --nn-preload default:GGML:AUTO:openchat-3.5-1210-Q5_K_M.gguf llama-api-server.wasm -p openchat -r '<|end_of_turn|>'
47
  ```
48
+
49
+ - Run as LlamaEdge command app
50
+
51
+ ```bash
52
+ wasmedge --dir .:. --nn-preload default:GGML:AUTO:openchat-3.5-1210-Q5_K_M.gguf llama-chat.wasm -p openchat -r '<|end_of_turn|>'
53
+ ```
54
+
55
+ ## Quantized GGUF Models
56
+
57
+ | Name | Quant method | Bits | Size | Use case |
58
+ | ---- | ---- | ---- | ---- | ----- |
59
+ | [openchat-3.5-1210-Q2_K.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q2_K.gguf) | Q2_K | 2 | 3.08 GB| smallest, significant quality loss - not recommended for most purposes |
60
+ | [openchat-3.5-1210-Q3_K_L.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q3_K_L.gguf) | Q3_K_L | 3 | 3.82 GB| small, substantial quality loss |
61
+ | [openchat-3.5-1210-Q3_K_M.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q3_K_M.gguf) | Q3_K_M | 3 | 3.52 GB| very small, high quality loss |
62
+ | [openchat-3.5-1210-Q3_K_S.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q3_K_S.gguf) | Q3_K_S | 3 | 3.16 GB| very small, high quality loss |
63
+ | [openchat-3.5-1210-Q4_0.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q4_0.gguf) | Q4_0 | 4 | 4.11 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
64
+ | [openchat-3.5-1210-Q4_K_M.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q4_K_M.gguf) | Q4_K_M | 4 | 4.37 GB| medium, balanced quality - recommended |
65
+ | [openchat-3.5-1210-Q4_K_S.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q4_K_S.gguf) | Q4_K_S | 4 | 4.14 GB| small, greater quality loss |
66
+ | [openchat-3.5-1210-Q5_0.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q5_0.gguf) | Q5_0 | 5 | 5.00 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
67
+ | [openchat-3.5-1210-Q5_K_M.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q5_K_M.gguf) | Q5_K_M | 5 | 5.13 GB| large, very low quality loss - recommended |
68
+ | [openchat-3.5-1210-Q5_K_S.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q5_K_S.gguf) | Q5_K_S | 5 | 5.00 GB| large, low quality loss - recommended |
69
+ | [openchat-3.5-1210-Q6_K.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q6_K.gguf) | Q6_K | 6 | 5.94 GB| very large, extremely low quality loss |
70
+ | [openchat-3.5-1210-Q8_0.gguf](https://huggingface.co/second-state/openchat-3.5-1210-GGUF/blob/main/openchat-3.5-1210-Q8_0.gguf) | Q8_0 | 8 | 7.70 GB| very large, extremely low quality loss - not recommended |
openchat-3.5-1210.Q5_K_M.gguf DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fbbb15ed13c630110b1790e0a0a621419fbf2f5564ead433c6ded9f8f196b95c
3
- size 5131421792