JeongwonChoi
commited on
Commit
•
d6f6457
1
Parent(s):
dff2d05
Update README.md
Browse files
README.md
CHANGED
@@ -1,88 +1,111 @@
|
|
1 |
---
|
2 |
tags:
|
3 |
-
- text-generation
|
4 |
license: cc-by-nc-sa-4.0
|
5 |
language:
|
6 |
-
- ko
|
7 |
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
8 |
pipeline_tag: text-generation
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
|
13 |
-
|
14 |
-
|
15 |
-
<|user|>
|
16 |
-
|
17 |
-
대한민국의 수도는 어디야?</s>
|
18 |
-
|
19 |
-
<|assistant|>
|
20 |
---
|
21 |
|
22 |
# **DataVortexTL-1.1B-v0.1**
|
23 |
-
<img src="./DataVortex.png" alt="DataVortex" style="height: 8em;">
|
24 |
|
25 |
-
|
26 |
-
|
27 |
-
[cc-by-nc-sa-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
|
28 |
|
29 |
## **Model Details**
|
30 |
|
31 |
### **Base Model**
|
32 |
-
|
|
|
33 |
|
34 |
### **Trained On**
|
35 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
36 |
|
37 |
### **Instruction format**
|
38 |
|
39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
40 |
|
41 |
## **Model Benchmark**
|
42 |
|
43 |
-
### **Ko-LLM-Leaderboard**
|
|
|
|
|
44 |
|
45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
46 |
|
47 |
-
|
48 |
|
49 |
-
|
50 |
You can use the code below.
|
51 |
|
52 |
```python
|
53 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
54 |
|
55 |
-
device = "cuda"
|
56 |
|
57 |
-
model = AutoModelForCausalLM.from_pretrained("Edentns/DataVortexTL-1.1B-v0.1"
|
58 |
tokenizer = AutoTokenizer.from_pretrained("Edentns/DataVortexTL-1.1B-v0.1")
|
59 |
|
60 |
messages = [
|
61 |
-
{
|
|
|
|
|
|
|
62 |
]
|
63 |
|
64 |
-
|
65 |
-
|
66 |
-
|
67 |
-
|
68 |
-
|
69 |
-
|
70 |
-
|
71 |
-
decoded
|
72 |
-
input_ids=encoded,
|
73 |
-
temperature=0.2,
|
74 |
-
top_p=0.9,
|
75 |
-
repetition_penalty=1.2,
|
76 |
-
do_sample=True,
|
77 |
-
max_length=4096,
|
78 |
-
eos_token_id=tokenizer.eos_token_id,
|
79 |
-
pad_token_id=tokenizer.eos_token_id
|
80 |
-
)
|
81 |
-
decoded = decoded[0][encoded.shape[1]:decoded[0].shape[-1]]
|
82 |
-
decoded_text = tokenizer.decode(decoded, skip_special_tokens=True)
|
83 |
-
print(decoded_text)
|
84 |
```
|
85 |
|
|
|
|
|
|
|
|
|
86 |
<div align="center">
|
87 |
<a href="https://edentns.com/">
|
88 |
<img src="./Logo.png" alt="Logo" style="height: 3em;">
|
|
|
1 |
---
|
2 |
tags:
|
3 |
+
- text-generation
|
4 |
license: cc-by-nc-sa-4.0
|
5 |
language:
|
6 |
+
- ko
|
7 |
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
8 |
pipeline_tag: text-generation
|
9 |
+
datasets:
|
10 |
+
- beomi/KoAlpaca-v1.1a
|
11 |
+
- jojo0217/korean_rlhf_dataset
|
12 |
+
- kyujinpy/OpenOrca-KO
|
13 |
+
- nlpai-lab/kullm-v2
|
|
|
|
|
|
|
|
|
|
|
|
|
14 |
---
|
15 |
|
16 |
# **DataVortexTL-1.1B-v0.1**
|
|
|
17 |
|
18 |
+
<img src="./DataVortex.png" alt="DataVortex" style="height: 8em;">
|
|
|
|
|
19 |
|
20 |
## **Model Details**
|
21 |
|
22 |
### **Base Model**
|
23 |
+
|
24 |
+
[TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
|
25 |
|
26 |
### **Trained On**
|
27 |
+
|
28 |
+
- **OS**: Ubuntu 20.04
|
29 |
+
- **GPU**: H100 80GB 1ea
|
30 |
+
- **transformers**: v4.36.2
|
31 |
+
|
32 |
+
### **Dataset**
|
33 |
+
|
34 |
+
- [beomi/KoAlpaca-v1.1a](https://huggingface.co/datasets/beomi/KoAlpaca-v1.1a)
|
35 |
+
- [jojo0217/korean_rlhf_dataset](https://huggingface.co/datasets/jojo0217/korean_rlhf_dataset)
|
36 |
+
- [kyujinpy/OpenOrca-KO](https://huggingface.co/datasets/kyujinpy/OpenOrca-KO)
|
37 |
+
- [nlpai-lab/kullm-v2](https://huggingface.co/datasets/nlpai-lab/kullm-v2)
|
38 |
|
39 |
### **Instruction format**
|
40 |
|
41 |
+
It follows **TinyLlama** format.
|
42 |
+
|
43 |
+
E.g.
|
44 |
+
|
45 |
+
```python
|
46 |
+
text = """\
|
47 |
+
<|system|>
|
48 |
+
당신은 사람들이 정보를 찾을 수 있도록 도와주는 인공지능 비서입니다.</s>
|
49 |
+
<|user|>
|
50 |
+
대한민국의 수도는 어디야?</s>
|
51 |
+
<|assistant|>
|
52 |
+
대한민국의 수도는 서울입니다.</s>
|
53 |
+
<|user|>
|
54 |
+
서울 인구는 총 몇 명이야?</s>
|
55 |
+
"""
|
56 |
+
```
|
57 |
|
58 |
## **Model Benchmark**
|
59 |
|
60 |
+
### **[Ko-LLM-Leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard)**
|
61 |
+
|
62 |
+
On Benchmarking ...
|
63 |
|
64 |
+
| Model | Average | Ko-ARC | Ko-HellaSwag | Ko-MMLU | Ko-TruthfulQA | Ko-CommonGen V2 |
|
65 |
+
| ---------------------------- | ------- | ------ | ------------ | ------- | ------------- | --------------- |
|
66 |
+
| DataVortexM-7B-Instruct-v0.1 | 39.81 | 34.13 | 42.35 | 38.73 | 45.46 | 38.37 |
|
67 |
+
| DataVortexS-10.7B-v0.1 | 0 | 0 | 0 | 0 | 0 | 0 |
|
68 |
+
| DataVortexS-10.7B-v0.2 | 43.6 | 38.74 | 50.74 | 38.98 | 44.7 | 44.86 |
|
69 |
+
| DataVortexS-10.7B-v0.3 | 0 | 0 | 0 | 0 | 0 | 0 |
|
70 |
+
| DataVortexS-10.7B-v0.4 | 0 | 0 | 0 | 0 | 0 | 0 |
|
71 |
+
| DataVortexS-10.7B-v0.4 | 0 | 0 | 0 | 0 | 0 | 0 |
|
72 |
+
| **DataVortexTL-1.1B-v0.1** | **0** | **0** | **0** | **0** | **0** | **0** |
|
73 |
+
| DataVortexS-10.7B-dpo-v0.1 | 0 | 0 | 0 | 0 | 0 | 0 |
|
74 |
|
75 |
+
## **Implementation Code**
|
76 |
|
77 |
+
This model contains the chat_template instruction format.
|
78 |
You can use the code below.
|
79 |
|
80 |
```python
|
81 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
82 |
|
83 |
+
device = "cuda" # the device to load the model onto
|
84 |
|
85 |
+
model = AutoModelForCausalLM.from_pretrained("Edentns/DataVortexTL-1.1B-v0.1")
|
86 |
tokenizer = AutoTokenizer.from_pretrained("Edentns/DataVortexTL-1.1B-v0.1")
|
87 |
|
88 |
messages = [
|
89 |
+
{"role": "system", "content": "당신은 사람들이 정보를 찾을 수 있도록 도와주는 인공지능 비서입니다."},
|
90 |
+
{"role": "user", "content": "대한민국의 수도는 어디야?"},
|
91 |
+
{"role": "assistant", "content": "대한민국의 수도는 서울입니다."},
|
92 |
+
{"role": "user", "content": "서울 인구는 총 몇 명이야?"}
|
93 |
]
|
94 |
|
95 |
+
encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
|
96 |
+
|
97 |
+
model_inputs = encodeds.to(device)
|
98 |
+
model.to(device)
|
99 |
+
|
100 |
+
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
|
101 |
+
decoded = tokenizer.batch_decode(generated_ids)
|
102 |
+
print(decoded[0])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
103 |
```
|
104 |
|
105 |
+
## **License**
|
106 |
+
|
107 |
+
The model is licensed under the [cc-by-nc-sa-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license, which allows others to copy, modify, and share the work non-commercially, as long as they give appropriate credit and distribute any derivative works under the same license.
|
108 |
+
|
109 |
<div align="center">
|
110 |
<a href="https://edentns.com/">
|
111 |
<img src="./Logo.png" alt="Logo" style="height: 3em;">
|