|
--- |
|
tags: |
|
- deepsparse |
|
--- |
|
## Usage |
|
|
|
```python |
|
from deepsparse import TextGeneration |
|
|
|
prompt = "How to make banana bread?" |
|
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n" |
|
|
|
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant") |
|
print(model(formatted_prompt, max_new_tokens=500).generations[0].text) |
|
|
|
""" |
|
Banana bread is a delicious and easy-to-make recipe that is sure to please. Here is a recipe for making banana bread: |
|
|
|
Ingredients: |
|
|
|
For the Banana Bread: |
|
|
|
- 1 cup of sugar |
|
- 1 cup of flour |
|
- 1/2 cup of mashed bananas |
|
- 1/4 cup of milk |
|
- 1/2 cup of melted butter |
|
- 1/4 cup of baking powder |
|
- 1/4 cup of baking soda |
|
- 1/4 cup of eggs |
|
- 1/4 cup of milk |
|
- 1/4 cup of sugar |
|
|
|
|
|
Instructions: |
|
|
|
1. Preheat the oven to 325°F (160°C). |
|
2. In a large bowl, combine the sugar and flour. |
|
3. In a separate bow, combine the mashed bananas, milk, butter, baking powder, baking soda, milk, sugar. |
|
4. Add the bananas and milk into the flour-sugar mixture. |
|
5. Pour the milk into the bowl of the flour-sugar mixture. |
|
6. Pour the baking powder into the bowl of the flour-sugar mixture. |
|
7. Pour the mashed bananas into the bowl of the flour-sugar mixture. |
|
8. Add the eggs into the bowl of the flour-sugar mixture. |
|
9. Stir the mixture until it becomes a dough. |
|
10. Grease a 9-inch (23 cm) square pan. |
|
11. Pour the mixture into the pan. |
|
12. Bake the banana bread in the oven for 40 minutes. |
|
13. Remove the banana bread from the oven and cool it. |
|
14. Cut the bread into 16 pieces. |
|
15. Make the glaze: |
|
16. Sprinkle the sugar over the bread. |
|
17. Bake the bread in the oven for 30 minutes. |
|
""" |
|
``` |
|
```python |
|
from deepsparse import TextGeneration |
|
|
|
prompt = "How to get in a good university?" |
|
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n" |
|
|
|
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant") |
|
print(model(formatted_prompt, max_new_tokens=200).generations[0].text) |
|
|
|
""" |
|
There are many factors to consider when choosing a university. Here are some tips for getting into a good university: |
|
|
|
1. Research your options: Consider the schools in your area and the ones in your desired location. Research their reputation, tuition, and academic programs. |
|
|
|
2. Apply to multiple universities: Apply to multiple universities, ensuring that you are applying to the best option for you. |
|
|
|
3. Get a job: If you are applying to a university, you will need to find a job to support your studies. This will help you budget and manage your time. |
|
|
|
4. Get involved with your community: Your university will likely have a community of students and faculty. Engage with this community by volunteering, participating in clubs, and engaging with others in your community. |
|
|
|
5. Get involved with extracurricular activities: Universities often have many extracurricular activities, which can help you meet new people |
|
""" |
|
``` |
|
## One-shot and Export |
|
|
|
```bash |
|
git clone https://github.com/neuralmagic/sparseml |
|
pip install -e "sparseml[transformers]" |
|
wget https://huggingface.co/nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant/raw/main/recipe.yaml # download recipe |
|
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v0.4 open_platypus --recipe recipe.yaml --save True |
|
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment |
|
cp deployment/model.onnx deployment/model-orig.onnx |
|
wget https://huggingface.co/nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant/raw/main/onnx_kv_inject.py # kv_cache file |
|
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx |
|
``` |
