|
--- |
|
tags: |
|
- deepsparse |
|
--- |
|
## Usage |
|
|
|
```python |
|
from deepsparse import TextGeneration |
|
|
|
prompt = "How to make banana bread?" |
|
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n" |
|
|
|
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant") |
|
print(model(formatted_prompt, max_new_tokens=300).generations[0].text) |
|
|
|
""" |
|
Banana bread is a delicious and healthy recipe that is easy to make. Here is a recipe for banana bread: |
|
|
|
Ingredients: |
|
|
|
1. 15 bananas |
|
2. 1 cup of mashed bananas |
|
3. 1 cup of milk |
|
4. 1 cup of eggs |
|
5. 1 cup of sugar |
|
6. 1 cup of flour |
|
7. 1 teaspoon of baking powder |
|
8. 1 teaspoon of baking sugar |
|
9. 1 cup of whole wheat flour |
|
10. 1 cup of whole wheat sugar |
|
11. 1 cup of whole wheat flour |
|
12. 1 cup of whole wheat flour |
|
13. 1 cup of whole wheat flour |
|
14. 1 cup of whole wheat flour |
|
15. 1 cup of whole wheat flour |
|
|
|
Step 1: Preheat the oven to 325°F (164°C). |
|
|
|
Step 2: Grease a 10- to 12-inch (25- to 35-centimeter) round baking pan. |
|
|
|
Step 3: Add the ingredients for the batter. |
|
|
|
Step 4: Pour the batter into the prepared pan. |
|
|
|
Step 5 |
|
""" |
|
``` |
|
```python |
|
from deepsparse import TextGeneration |
|
|
|
prompt = "How to get in a good university?" |
|
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n" |
|
|
|
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant") |
|
print(model(formatted_prompt, max_new_tokens=200).generations[0].text) |
|
|
|
""" |
|
There are several factors to consider when choosing a university: |
|
|
|
1. Location: Where is the university located? Is it close to a city where you want to live? |
|
2. Faculty: What faculty members are teaching at the university? Is there a specialty you want to pursue? |
|
3. Tuition: How much does the university cost? |
|
4. Amenities: What amenities does the university offer? Is there a library, a gym, or other resources? |
|
5. Communication: How easy is it to get to campus? Is there a campus-wide bus system? |
|
6. Jobs: What jobs are available at the university? Is there a job market? |
|
|
|
It's important to consider the location, faculty, and amenities when choosing a university. It's also important to research the job market and the salaries for jobs in the university. It's also important to research the job market and the |
|
""" |
|
``` |
|
## One-shot and Export |
|
|
|
```bash |
|
git clone https://github.com/neuralmagic/sparseml |
|
pip install -e "sparseml[transformers]" "torch<2" |
|
cd sparseml |
|
git checkout update/onnx_export/duplicate |
|
python src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v0.4 open_platypus --recipe recipe.yaml --save True |
|
python src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment |
|
cp deployment/model.onnx deployment/model-orig.onnx |
|
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx |
|
``` |
