Update README.md
Browse files
README.md
CHANGED
@@ -24,6 +24,30 @@ The following models were included in the merge:
|
|
24 |
* [sethuiyer/Medichat-Llama3-8B](https://huggingface.co/sethuiyer/Medichat-Llama3-8B)
|
25 |
* [johnsnowlabs/JSL-MedLlama-3-8B-v2.0](https://huggingface.co/johnsnowlabs/JSL-MedLlama-3-8B-v2.0)
|
26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
27 |
### Configuration
|
28 |
|
29 |
The following YAML configuration was used to produce this model:
|
|
|
24 |
* [sethuiyer/Medichat-Llama3-8B](https://huggingface.co/sethuiyer/Medichat-Llama3-8B)
|
25 |
* [johnsnowlabs/JSL-MedLlama-3-8B-v2.0](https://huggingface.co/johnsnowlabs/JSL-MedLlama-3-8B-v2.0)
|
26 |
|
27 |
+
### Evaluation
|
28 |
+
|
29 |
+
- multimedq (0 shot)
|
30 |
+
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|
31 |
+
|-------------------------------|-------|------|-----:|--------|-----:|---|-----:|
|
32 |
+
| - medmcqa |Yaml |none | 0|acc |0.5728|± |0.0076|
|
33 |
+
| | |none | 0|acc_norm|0.5728|± |0.0076|
|
34 |
+
| - medqa_4options |Yaml |none | 0|acc |0.5923|± |0.0138|
|
35 |
+
| | |none | 0|acc_norm|0.5923|± |0.0138|
|
36 |
+
| - anatomy (mmlu) | 0|none | 0|acc |0.7111|± |0.0392|
|
37 |
+
| - clinical_knowledge (mmlu) | 0|none | 0|acc |0.7547|± |0.0265|
|
38 |
+
| - college_biology (mmlu) | 0|none | 0|acc |0.7917|± |0.0340|
|
39 |
+
| - college_medicine (mmlu) | 0|none | 0|acc |0.6647|± |0.0360|
|
40 |
+
| - medical_genetics (mmlu) | 0|none | 0|acc |0.8200|± |0.0386|
|
41 |
+
| - professional_medicine (mmlu)| 0|none | 0|acc |0.7426|± |0.0266|
|
42 |
+
|stem |N/A |none | 0|acc_norm|0.5773|± |0.0067|
|
43 |
+
| | |none | 0|acc |0.6145|± |0.0057|
|
44 |
+
| - pubmedqa | 1|none | 0|acc |0.7400|± |0.0196|
|
45 |
+
|
46 |
+
|Groups|Version|Filter|n-shot| Metric |Value | |Stderr|
|
47 |
+
|------|-------|------|-----:|--------|-----:|---|-----:|
|
48 |
+
|stem |N/A |none | 0|acc_norm|0.5773|± |0.0067|
|
49 |
+
| | |none | 0|acc |0.6145|± |0.0057|
|
50 |
+
|
51 |
### Configuration
|
52 |
|
53 |
The following YAML configuration was used to produce this model:
|