File size: 3,994 Bytes
33c7513 ef88267 33c7513 05be2ea 33c7513 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
---
base_model: unsloth/DeepSeek-R1-Distill-Qwen-1.5B-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- qwen2
- trl
- sft
license: apache-2.0
language:
- en
---
Function calling requires two step inferences, below is the example:
# Step 1:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import json
model_id = "R1_tool_call_Distill-Qwen-1.5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
tools = [
{
"name": "create_contact",
"description": "Create a new contact",
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The name of the contact"
},
"email": {
"type": "string",
"description": "The email address of the contact"
}
},
"required": ["name", "email"]
}
}
]
messages = [
{ "role": "user", "content": f"""You have access to these tools, use them if necessary: {tools}
I need to create a new contact for my friend John Doe. His email is [email protected]."""
}
]
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
outputs = model.generate(
input_ids,
max_new_tokens=256,
do_sample=True,
temperature=0.6,
top_p=0.9,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
# >>
# >> <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>create_contact
# >> ```json
# >> {"name": "John Doe", "email": "[email protected]"}
# >> ```<｜tool▁call▁end｜><｜tool▁calls▁end｜>
# Above is a response from assistant, you need to parse it and execute a tool on your own.
```
# Step 2:
```python
messages = [
{"role": "user", "content": """You have access to these tools, use them if necessary: {tools}\n\nI need to create a new contact for my friend John Doe. His email is [email protected]."""},
{"role": "assistant", "content": None, "tool_calls": [
{
"type": "function",
"function": {
"name": "create_contact",
"arguments": json.dumps({"name": "John Doe", "email": "[email protected]"})
}
},
]},
{"role": "tool", "name": "create_contact", "content": """{"status": "success", "message": "Contact for John Doe with email [email protected] has been created successfully."}"""},
]
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
outputs = model.generate(
input_ids,
max_new_tokens=256,
do_sample=True,
temperature=0.6,
top_p=0.9,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
# >>
# >> <think>Based on the user's request, I have created a contact for John Doe with his email address. The tool has successfully created the contact. I will now provide the contact information to the user.</think>
# >> The contact for John Doe has been successfully created with the email address [email protected]. Please feel free to reach out to him if needed.
```
# Limitations:
- The model sometimes refused not to think
# Uploaded model
- **Developed by:** hiieu
- **License:** apache-2.0
- **Finetuned from model :** unsloth/DeepSeek-R1-Distill-Qwen-1.5B-unsloth-bnb-4bit
This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|