|
--- |
|
tags: |
|
- deepsparse |
|
--- |
|
## Usage |
|
|
|
```python |
|
from deepsparse import TextGeneration |
|
|
|
prompt = "How to get in a good university?" |
|
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n" |
|
|
|
model = TextGeneration(model="hf:nm-testing/MiniChat-3B-pruned-quant-50") |
|
|
|
print(model(formatted_prompt, max_new_tokens=500).generations[0].text) |
|
""" |
|
Hi there! I'm an AI assistant and I can help you with any questions you have. Here's how you can get into a good university: |
|
1. Choose a university that you are interested in. |
|
2. Read the university's website and learn more about their programs and courses. |
|
3. Reach out to the university's alumni or staff members and ask for advice or guidance. |
|
4. Attend a campus tour or visit day to get a better understanding of the university's culture and atmosphere. |
|
5. Finally, don't wait to apply for the university of your choice. |
|
I hope this helps you get into a good university! Let me know if you have any other questions or need any more information. |
|
""" |
|
``` |
|
```python |
|
from deepsparse import TextGeneration |
|
|
|
prompt = "How to become a great software engineer?" |
|
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n" |
|
|
|
model = TextGeneration(model="hf:nm-testing/MiniChat-3B-pruned-quant-50") |
|
|
|
print(model(formatted_prompt, max_new_tokens=500).generations[0].text) |
|
""" |
|
1. Learn the basics of programming languages like C++, Python, and Java. |
|
2. Understand the fundamentals of computer science and programming concepts. |
|
3. Practice and experiment with different programming languages and techniques. |
|
4. Continuously update your knowledge and skills to keep up with the latest technologies. |
|
5. Become familiar with the latest tools and technologies like Git, Docker, and virtual reality (VR) |
|
6. Continuously seek opportunities to learn and grow in your career. |
|
7. Become a part of open-source communities and contribute to open-source projects. |
|
8. Continuously seek opportunities to learn and grow in your career. |
|
9. Continuously seek opportunities to learn and grow in your career. |
|
10. Continuously seek opportunities to learn and grow in your career. |
|
""" |
|
|
|
``` |
|
## One-shot and Export |
|
|
|
```bash |
|
git clone https://github.com/neuralmagic/sparseml |
|
pip install -e "sparseml[transformers]" |
|
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-3B open_platypus --recipe recipe.yaml --save True |
|
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment |
|
cp deployment/model.onnx deployment/model-orig.onnx |
|
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx |
|
``` |
