function call for llm inference
I am trying to use tools with the mixtral model. One tool I am trying to design is a note taker that can capture important information from a piece of text that will be useful to answer a user question. However, I am not sure if I am going about it correctly as I just get back the same input as my final output:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
from huggingface_hub import InferenceClient
def note_taker(text: str):
"""
A function that takes a user question and a context text, and generates notes on the relevant details to help answer the user's question.
Args:
text: string containing user question and context
Returns:
Notes on the relevant details to help answer the user's question, as a string.
"""
client = InferenceClient(model="<tgi_url>")
prompt = f"Generate notes on the relevant details from the provide context to answer the user's question. \n\nText:{text} \n Notes: "
notes = client.text_generation(prompt=prompt)
return notes
tool_use_prompt = """<s>[AVAILABLE_TOOLS] [{"type": "function", "function": {"name": "note_taker", "description": "A function that takes a user question and a context text, and generates notes on the relevant details to help answer the user's question.", "parameters": {"type": "object", "properties": {"text": {"type": "string", "description": "string containing user question and context"}}, "required": ["text"]}}}][/AVAILABLE_TOOLS][INST] Question: What is the capital of France?
Context: France is a country in Europe. The capital of France is Paris. It is known for its art, culture, and history[/INST]"""
client = InferenceClient(model="<mixtral_tgi_url>")
output_text = client.text_generation(prompt=tool_use_prompt)
print(f"Output: {output_text}")
Output: [{"name": "note_taker", "arguments": {"text": "Question: What is the capital of France? Context: France is a country in Europe. The capital of France is Paris. It is known for its art, culture, and history"}}]
Any pointers are appreciated, thank you!
the question is what are the ote being taken ? what is interestig information ?
What are keypoints to rememeber?
Ok the capital of a country is A or B but what is the relevance and importance of this data !
So for note taking we need a strategy !
as
EIther your copy all
or
just some key points based on or learning objective :
So maybe All questions which contain (who/What/Why/Where/How) ?
If we were taking notes for a specfic topic we n=may just choose all sentences which contain a specific keyword pack :
So we would have an entity list , based on your expected topics . and any phrases which contain any one of these keywords ... then we shoudl take that note !
So in the end we would have keyphrases associated with that topic as well as ay Who/Why/What questions and answers !<<
We may also choose Nouns ! ? anyPhrase which contains a proNoun ? this would save the above text as a not becuse it falls in all of these niches~
So Once you have a Strategey for generalized note taking then your desription in the docsrrings should be enough to trigger your tools to take a note !
i hope this pointyou in the right direction ( you =may want to capture all events and locations too ! , Times and dates ) <<<< Extending the strategy further :