Update README.md
Browse files
README.md
CHANGED
|
@@ -60,7 +60,24 @@ For an example of a roleplaying discord bot, check out this: https://github.com/
|
|
| 60 |
The model is currently being uploaded in FP16 format, and there are plans to convert the model to GGML and GPTQ 4bit quantizations. The team is also working on a full benchmark, similar to what was done for GPT4-x-Vicuna. We will try to get in discussions to get the model included in the GPT4All.
|
| 61 |
|
| 62 |
## Benchmark Results
|
| 63 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
## Model Usage
|
| 66 |
The model is available for download on Hugging Face. It is suitable for a wide range of language tasks, from generating creative text to understanding and following complex instructions.
|
|
|
|
| 60 |
The model is currently being uploaded in FP16 format, and there are plans to convert the model to GGML and GPTQ 4bit quantizations. The team is also working on a full benchmark, similar to what was done for GPT4-x-Vicuna. We will try to get in discussions to get the model included in the GPT4All.
|
| 61 |
|
| 62 |
## Benchmark Results
|
| 63 |
+
```
|
| 64 |
+
| Task |Version| Metric |Value | |Stderr|
|
| 65 |
+
|-------------|------:|--------|-----:|---|-----:|
|
| 66 |
+
|arc_challenge| 0|acc |0.4915|± |0.0146|
|
| 67 |
+
| | |acc_norm|0.5085|± |0.0146|
|
| 68 |
+
|arc_easy | 0|acc |0.7769|± |0.0085|
|
| 69 |
+
| | |acc_norm|0.7424|± |0.0090|
|
| 70 |
+
|boolq | 1|acc |0.7948|± |0.0071|
|
| 71 |
+
|hellaswag | 0|acc |0.6143|± |0.0049|
|
| 72 |
+
| | |acc_norm|0.8000|± |0.0040|
|
| 73 |
+
|openbookqa | 0|acc |0.3560|± |0.0214|
|
| 74 |
+
| | |acc_norm|0.4640|± |0.0223|
|
| 75 |
+
|piqa | 0|acc |0.7965|± |0.0094|
|
| 76 |
+
| | |acc_norm|0.7889|± |0.0095|
|
| 77 |
+
|winogrande | 0|acc |0.7190|± |0.0126|
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
These benchmarks currently have us at #1 on ARC-c, ARC-e, Hellaswag, and OpenBookQA, and 2nd place on Winogrande, comparing to GPT4all's benchmarking list.
|
| 81 |
|
| 82 |
## Model Usage
|
| 83 |
The model is available for download on Hugging Face. It is suitable for a wide range of language tasks, from generating creative text to understanding and following complex instructions.
|