The purpose of generating a Function description
I tried to use the model for inference, and it can output the corresponding content normally, but I have some questions about the function description, definition part in the second half of the output:
When outputting <nexa_0>('front')<nexa_end>, the program should know which pre-trained function to call, so why do I need to output the Function description part, which looks like it will only increase the inference time.
I'm not sure if there are some details I'm overlooking, looking forward to your replies.
==========
Prompt: Below is the query from the users, please call the correct function and generate the parameters to call the function.\n\nQuery: Take a selfie for me with front camera \n\nResponse:
<nexa_0>('front')<nexa_end>
Function description:
def take_a_photo(camera):
"""
Captures a photo using the specified camera and resolution settings.
Parameters:
- camera (str): Specifies the camera to use. Can be 'front' or 'back'. The default is 'back'.
Returns:
- str: The string contains the file path of the captured photo if successful, or an error message if not. Example: '/storage/emulated/0/Pictures/MyApp/IMG_20240310_123456.jpg'
"""
==========
Prompt: 342.046 tokens-per-sec
Generation: 59.512 tokens-per-sec
You shouldn't generate that. You should use <nexa_end> as the early stopping criterion. This is included in the paper, and it will make inference faster.
You shouldn't generate that. You should use
<nexa_end>as the early stopping criterion. This is included in the paper, and it will make inference faster.
Hi~ Does this mean that the input during training is:
"Below is the query from the users, please call the correct function and generate the parameters to call the function.\n\nQuery: Take a selfie for me with front camera \n\nResponse:",
and the output during training is:
" <nexa_0>('front')<nexa_end>
Function description:
def take_a_photo(camera):
"""
Captures a photo using the specified camera and resolution settings.
Parameters:
- camera (str): Specifies the camera to use. Can be 'front' or 'back'. The default is 'back'.
Returns:
- str: The string contains the file path of the captured photo if successful, or an error message if not. Example: '/storage/emulated/0/Pictures/MyApp/IMG_20240310_123456.jpg'
"""
" ?
@xieyf Here is one example, to set up early stopping, use eos_token_id :
outputs = model.generate(
input_ids=input_ids["input_ids"],
max_length=1024,
eos_token_id=256021,
do_sample=False)