|
--- |
|
license: mit |
|
datasets: |
|
- wenbopan/Fusang-v1 |
|
- wenbopan/OpenOrca-zh-20k |
|
language: |
|
- zh |
|
- en |
|
--- |
|
|
|
 |
|
|
|
**The Faro chat model focuses on practicality and long-context modeling. It handles various downstream tasks with higher quality, delivering stable and reliable results even when inputs contain lengthy documents or complex instructions. Faro seamlessly works in both English and Chinese.** |
|
|
|
# Faro-Yi-34B-200K |
|
Faro-Yi-34B-200K is an improved [Yi-34B-200K](https://huggingface.co/01-ai/Yi-34B-200K) with extensive instruction tuning on [Fusang-V1](https://huggingface.co/datasets/wenbopan/Fusang-v1). Compared to Yi-34B-200K, Faro-Yi-34B-200K has gained greater capability in various downstream tasks and long-context modeling thanks to the large-scale synthetic data in Fusang-V1. |
|
|
|
## How to Use |
|
|
|
Faro-Yi-34B-200K uses chatml template. This make it easy to set up system prompt and multi-turn conversations. It truly excels when used for analyzing long documents or instructions. I recommend using vLLM for long inputs. |
|
|
|
```python |
|
import io |
|
import requests |
|
from PyPDF2 import PdfReader |
|
from vllm import LLM, SamplingParams |
|
|
|
llm = LLM(model="wenbopan/Faro-Yi-34B-200K") |
|
|
|
pdf_data = io.BytesIO(requests.get("https://arxiv.org/pdf/2303.08774.pdf").content) |
|
document = "".join(page.extract_text() for page in PdfReader(pdf_data).pages) # 100 pages |
|
|
|
question = f"{document}\n\nAccording to the paper, what is the parameter count of GPT-4?" |
|
messages = [ {"role": "user", "content": question} ] # 83K tokens |
|
prompt = llm.get_tokenizer().apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
|
output = llm.generate(prompt, SamplingParams(temperature=0.8, max_tokens=500)) |
|
print(output[0].outputs[0].text) |
|
# Yi-9B-200K: 175B. GPT-4 has 175B \nparameters. How many models were combined to create GPT-4? Answer: 6. ... |
|
# Faro-Yi-9B-200K: GPT-4 does not have a publicly disclosed parameter count due to the competitive landscape and safety implications of large-scale models like GPT-4. ... |
|
``` |
|
|
|
<details> <summary>Or With Transformers</summary> |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained('wenbopan/Faro-Yi-34B-200K', device_map="cuda") |
|
tokenizer = AutoTokenizer.from_pretrained('wenbopan/Faro-Yi-34B-200K') |
|
messages = [ |
|
{"role": "system", "content": "You are a helpful assistant. Always answer with a short response."}, |
|
{"role": "user", "content": "Tell me what is Pythagorean theorem like you are a pirate."} |
|
] |
|
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device) |
|
generated_ids = model.generate(input_ids, max_new_tokens=512, temperature=0.5) |
|
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True) # Aye, matey! The Pythagorean theorem is a nautical rule that helps us find the length of the third side of a triangle. ... |
|
``` |
|
|
|
</details> |
