Hugging Face's logo

openthaigpt
/
openthaigpt1.5-14b-instruct

Text Generation
Transformers
Safetensors
GGUF
Thai
English
qwen2
openthaigpt
qwen
conversational
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
1
kobkrit commited on
Commit
872a756
·
verified ·
1 Parent(s): 959f7b0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -22
README.md CHANGED
@@ -32,31 +32,31 @@ https://github.com/OpenThaiGPT/openthaigpt1.5_api_examples
32
 
33
  ## Benchmark on [OpenThaiGPT Eval](https://huggingface.co/datasets/openthaigpt/openthaigpt_eval)
34
  ** Please take a look at ``openthaigpt/openthaigpt1.5-7b-instruct`` for this model's evaluation result.
35
- | **Exam names** | **scb10x/llama-3-typhoon-v1.5x-8b-instruct** | **meta-llama/Llama-3.1-7B-Instruct** | **Qwen/Qwen2.5-7B-Instruct_stat** | **openthaigpt/openthaigpt1.5-7b** |
36
- |:------------------------------:|:--------------------------------------------:|:------------------------------------:|:---------------------------------:|:---------------------------------:|
37
- | **01_a_level** | 46.67% | 47.50% | 58.33% | 60.00% |
38
- | **02_tgat** | 32.00% | 36.00% | 32.00% | 36.00% |
39
- | **03_tpat1** | 52.50% | 55.00% | 57.50% | 57.50% |
40
- | **04_investment_consult** | 56.00% | 48.00% | 68.00% | 76.00% |
41
- | **05_facebook_beleble_th_200** | 78.00% | 73.00% | 79.00% | 81.00% |
42
- | **06_xcopa_th_200** | 79.50% | 69.00% | 80.50% | 81.00% |
43
- | **07_xnli2.0_th_200** | 56.50% | 55.00% | 53.00% | 54.50% |
44
- | **08_onet_m3_thai** | 48.00% | 32.00% | 72.00% | 64.00% |
45
- | **09_onet_m3_social** | 75.00% | 50.00% | 90.00% | 80.00% |
46
- | **10_onet_m3_math** | 25.00% | 18.75% | 31.25% | 31.25% |
47
- | **11_onet_m3_science** | 46.15% | 42.31% | 46.15% | 46.15% |
48
- | **12_onet_m3_english** | 70.00% | 76.67% | 86.67% | 83.33% |
49
- | **13_onet_m6_thai** | 47.69% | 29.23% | 46.15% | 53.85% |
50
- | **14_onet_m6_math** | 29.41% | 17.65% | 29.41% | 29.41% |
51
- | **15_onet_m6_social** | 50.91% | 43.64% | 56.36% | 58.18% |
52
- | **16_onet_m6_science** | 42.86% | 32.14% | 57.14% | 57.14% |
53
- | **17_onet_m6_english** | 65.38% | 71.15% | 78.85% | 80.77% |
54
- | **Micro Average** | 60.65% | 55.60% | 64.41% | <b style="color:blue">65.78%</b> |
55
 
56
 
57
  Thai language multiple choice exams, Test on unseen test set, Zero-shot learning. Benchmark source code and exams information: https://github.com/OpenThaiGPT/openthaigpt_eval
58
 
59
- (Updated on: 30 September 2024)
60
 
61
  ## Benchmark on [scb10x/thai_exam](https://huggingface.co/datasets/scb10x/thai_exam)
62
 
@@ -315,7 +315,7 @@ print(response)
315
  | **Number of Parameters** | **FP 16 bits** | **8 bits (Quantized)** | **4 bits (Quantized)** | **Example Graphic Card for 4 bits** |
316
  |------------------|----------------|------------------------|------------------------|---------------------------------------------|
317
  | **7b** | 24 GB | 12 GB | 6 GB | Nvidia RTX 4060 8GB |
318
- | **13b** | 48 GB | 24 GB | 12 GB | Nvidia RTX 4070 16GB |
319
  | **72b** | 192 GB | 96 GB | 48 GB | Nvidia RTX 4090 24GB x 2 cards |
320
 
321
  ### Authors
 
32
 
33
  ## Benchmark on [OpenThaiGPT Eval](https://huggingface.co/datasets/openthaigpt/openthaigpt_eval)
34
  ** Please take a look at ``openthaigpt/openthaigpt1.5-7b-instruct`` for this model's evaluation result.
35
+ | **Exam names** | **openthaigpt/openthaigpt1.5-7b** | **openthaigpt/openthaigpt1.5-14b** | **openthaigpt/openthaigpt1.5-72b** |
36
+ |--------------------------------|-----------------------------------|------------------------------------|------------------------------------|
37
+ | **01_a_level** | 60.00% | 65.00% | 76.67% |
38
+ | **02_tgat** | 36.00% | 50.00% | 46.00% |
39
+ | **03_tpat1** | 57.50% | 52.50% | 55.00% |
40
+ | **04_investment_consult** | 76.00% | 72.00% | 72.00% |
41
+ | **05_facebook_beleble_th_200** | 81.00% | 87.00% | 90.00% |
42
+ | **06_xcopa_th_200** | 81.00% | 86.50% | 90.50% |
43
+ | **07_xnli2.0_th_200** | 54.50% | 64.50% | 70.50% |
44
+ | **08_onet_m3_thai** | 64.00% | 84.00% | 84.00% |
45
+ | **09_onet_m3_social** | 80.00% | 90.00% | 95.00% |
46
+ | **10_onet_m3_math** | 31.25% | 12.50% | 37.50% |
47
+ | **11_onet_m3_science** | 46.15% | 53.85% | 73.08% |
48
+ | **12_onet_m3_english** | 83.33% | 93.33% | 96.67% |
49
+ | **13_onet_m6_thai** | 53.85% | 56.92% | 56.92% |
50
+ | **14_onet_m6_math** | 29.41% | 41.18% | 41.18% |
51
+ | **15_onet_m6_social** | 58.18% | 61.82% | 65.45% |
52
+ | **16_onet_m6_science** | 57.14% | 57.14% | 67.86% |
53
+ | **17_onet_m6_english** | 80.77% | 78.85% | 90.38% |
54
+ | **Micro Average** | 65.78% | <b style="color:blue">71.51%</b> | 76.73% |
55
 
56
 
57
  Thai language multiple choice exams, Test on unseen test set, Zero-shot learning. Benchmark source code and exams information: https://github.com/OpenThaiGPT/openthaigpt_eval
58
 
59
+ (Updated on: 13 October 2024)
60
 
61
  ## Benchmark on [scb10x/thai_exam](https://huggingface.co/datasets/scb10x/thai_exam)
62
 
 
315
  | **Number of Parameters** | **FP 16 bits** | **8 bits (Quantized)** | **4 bits (Quantized)** | **Example Graphic Card for 4 bits** |
316
  |------------------|----------------|------------------------|------------------------|---------------------------------------------|
317
  | **7b** | 24 GB | 12 GB | 6 GB | Nvidia RTX 4060 8GB |
318
+ | **14b** | 48 GB | 24 GB | 12 GB | Nvidia RTX 4070 16GB |
319
  | **72b** | 192 GB | 96 GB | 48 GB | Nvidia RTX 4090 24GB x 2 cards |
320
 
321
  ### Authors