metadata
license: apache-2.0
pipeline_tag: text-generation
language:
- en
widget:
- example_title: Hello
messages:
- role: user
content: Hey my name is Julien! How are you?
- example_title: Winter holidays
messages:
- role: system
content: >-
You are a helpful and honest assistant. Please, respond concisely and
truthfully.
- role: user
content: Can you recommend a good destination for Winter holidays?
- example_title: Programming assistant
messages:
- role: system
content: >-
You are a helpful and honest code and programming assistant. Please,
respond concisely and truthfully.
- role: user
content: Write a function that computes the nth fibonacci number.
inference:
parameters:
max_new_tokens: 100
stop:
- <|end_of_text|>
- <|eot_id|>
WORK IN PROGESS
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.2193 | ± | 0.0121 |
| none | 0 | acc_norm | ↑ | 0.2517 | ± | 0.0127 | ||
| arc_easy | 1 | none | 0 | acc | ↑ | 0.2399 | ± | 0.0088 |
| none | 0 | acc_norm | ↑ | 0.2428 | ± | 0.0088 | ||
| boolq | 2 | none | 0 | acc | ↑ | 0.6116 | ± | 0.0085 |
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2546 | ± | 0.0043 |
| none | 0 | acc_norm | ↑ | 0.2647 | ± | 0.0044 | ||
| openbookqa | 1 | none | 0 | acc | ↑ | 0.1540 | ± | 0.0162 |
| none | 0 | acc_norm | ↑ | 0.2680 | ± | 0.0198 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.5413 | ± | 0.0116 |
| none | 0 | acc_norm | ↑ | 0.5310 | ± | 0.0116 | ||
| winogrande | 1 | none | 0 | acc | ↑ | 0.5020 | ± | 0.0141 |