|
--- |
|
library_name: transformers |
|
language: |
|
- en |
|
license: apache-2.0 |
|
base_model: pszemraj/tFINE-850m-24x24-v0.4-flan_aug |
|
metrics: |
|
- rouge |
|
datasets: |
|
- pszemraj/infinity-instruct-7m-T2T_en |
|
pipeline_tag: text2text-generation |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# tFINE-850m-24x24-v0.5-instruct-L1 |
|
|
|
This model is a fine-tuned version of [pszemraj/tFINE-850m-24x24-v0.4-flan_aug](https://huggingface.co/pszemraj/tFINE-850m-24x24-v0.4-flan_aug) on the pszemraj/infinity-instruct-7m-T2T_en dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 1.1478 |
|
- Rouge1: 38.4805 |
|
- Rouge2: 22.5971 |
|
- Rougel: 31.1093 |
|
- Rougelsum: 36.596 |
|
- Gen Len: 441.475 |
|
- Num Input Tokens Seen: 435513684 |
|
|
|
## usage |
|
|
|
```py |
|
from transformers import pipeline |
|
|
|
pipe = pipeline( |
|
"text2text-generation", |
|
model="pszemraj/tFINE-850m-24x24-v0.5-instruct-L1", |
|
device_map="auto", |
|
) |
|
prompt = "write a python script to download a file from a url and save as a local file using requests. explain how it works" |
|
res = pipe( |
|
prompt, |
|
max_new_tokens=192, |
|
top_k=4, |
|
penalty_alpha=0.6, |
|
renormalize_logits=True, |
|
no_repeat_ngram_size=5, |
|
) |
|
print(res[0]["generated_text"]) |
|
``` |
|
|
|
## Quick eval |
|
|
|
Quick eval for: `pszemraj/tFINE-850m-24x24-v0.5-instruct-L1` |
|
|
|
|
|
hf (pretrained=pszemraj/tFINE-850m-24x24-v0.5-instruct-L1,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8 |
|
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| |
|
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------| |
|
|boolq | 2|none | 0|acc |↑ |0.5661|± |0.0087| |
|
|openbookqa | 1|none | 0|acc |↑ |0.1540|± |0.0162| |
|
| | |none | 0|acc_norm |↑ |0.2960|± |0.0204| |
|
|piqa | 1|none | 0|acc |↑ |0.6094|± |0.0114| |
|
| | |none | 0|acc_norm |↑ |0.5952|± |0.0115| |
|
|social_iqa | 0|none | 0|acc |↑ |0.3900|± |0.0110| |
|
|tinyArc | 0|none | 25|acc_norm |↑ |0.2903|± | N/A| |
|
|tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.0471|± | N/A| |
|
| | |strict-match | 5|exact_match|↑ |0.0339|± | N/A| |
|
|tinyHellaswag| 0|none | 10|acc_norm |↑ |0.2490|± | N/A| |
|
|tinyMMLU | 0|none | 0|acc_norm |↑ |0.3021|± | N/A| |
|
|winogrande | 1|none | 0|acc |↑ |0.4925|± |0.0141| |
|
|
|
|
