File size: 2,874 Bytes
3409c51 d9ab5da 3409c51 d9ab5da 05a0ae8 3409c51 05a0ae8 d9ab5da 05a0ae8 d9ab5da 3409c51 d9ab5da 3409c51 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
---
tags:
- deepsparse
---
## Usage
```python
from deepsparse import TextGeneration
prompt = "How to make banana bread?"
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant")
print(model(formatted_prompt, max_new_tokens=300).generations[0].text)
"""
Banana bread is a delicious and healthy recipe that is easy to make. Here is a recipe for banana bread:
Ingredients:
1. 15 bananas
2. 1 cup of mashed bananas
3. 1 cup of milk
4. 1 cup of eggs
5. 1 cup of sugar
6. 1 cup of flour
7. 1 teaspoon of baking powder
8. 1 teaspoon of baking sugar
9. 1 cup of whole wheat flour
10. 1 cup of whole wheat sugar
11. 1 cup of whole wheat flour
12. 1 cup of whole wheat flour
13. 1 cup of whole wheat flour
14. 1 cup of whole wheat flour
15. 1 cup of whole wheat flour
Step 1: Preheat the oven to 325°F (164°C).
Step 2: Grease a 10- to 12-inch (25- to 35-centimeter) round baking pan.
Step 3: Add the ingredients for the batter.
Step 4: Pour the batter into the prepared pan.
Step 5
"""
```
```python
from deepsparse import TextGeneration
prompt = "How to get in a good university?"
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant")
print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
"""
There are several factors to consider when choosing a university:
1. Location: Where is the university located? Is it close to a city where you want to live?
2. Faculty: What faculty members are teaching at the university? Is there a specialty you want to pursue?
3. Tuition: How much does the university cost?
4. Amenities: What amenities does the university offer? Is there a library, a gym, or other resources?
5. Communication: How easy is it to get to campus? Is there a campus-wide bus system?
6. Jobs: What jobs are available at the university? Is there a job market?
It's important to consider the location, faculty, and amenities when choosing a university. It's also important to research the job market and the salaries for jobs in the university. It's also important to research the job market and the
"""
```
## One-shot and Export
```bash
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]" "torch<2"
cd sparseml
git checkout update/onnx_export/duplicate
python src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v0.4 open_platypus --recipe recipe.yaml --save True
python src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx
``` |