metadata
tags:
- deepsparse
Usage
from deepsparse import TextGeneration
prompt = "How to get in a good university?"
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n"
model = TextGeneration(model="hf:nm-testing/MiniChat-3B-pruned-quant-50")
print(model(formatted_prompt, max_new_tokens=500).generations[0].text)
"""
Hi there! I'm an AI assistant and I can help you with any questions you have. Here's how you can get into a good university:
1. Choose a university that you are interested in.
2. Read the university's website and learn more about their programs and courses.
3. Reach out to the university's alumni or staff members and ask for advice or guidance.
4. Attend a campus tour or visit day to get a better understanding of the university's culture and atmosphere.
5. Finally, don't wait to apply for the university of your choice.
I hope this helps you get into a good university! Let me know if you have any other questions or need any more information.
"""
from deepsparse import TextGeneration
prompt = "How to become a great software engineer?"
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n"
model = TextGeneration(model="hf:nm-testing/MiniChat-3B-pruned-quant-50")
print(model(formatted_prompt, max_new_tokens=500).generations[0].text)
"""
1. Learn the basics of programming languages like C++, Python, and Java.
2. Understand the fundamentals of computer science and programming concepts.
3. Practice and experiment with different programming languages and techniques.
4. Continuously update your knowledge and skills to keep up with the latest technologies.
5. Become familiar with the latest tools and technologies like Git, Docker, and virtual reality (VR)
6. Continuously seek opportunities to learn and grow in your career.
7. Become a part of open-source communities and contribute to open-source projects.
8. Continuously seek opportunities to learn and grow in your career.
9. Continuously seek opportunities to learn and grow in your career.
10. Continuously seek opportunities to learn and grow in your career.
"""
One-shot and Export
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-3B open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx