dolly v2 12 b performing poorly at Summarization task
My config - do_sample: bool = True, max_new_tokens: int = 1024, top_p: float = 0.92, top_k: int = 0
I'm using the v2 model as-is without any training. My objective is get information about a particular entity from the text. Hence the prompt.
prompt = """
Summarize about Intel from the following meeting notes.
A discussion about sourcing part PN569 for the company's project took place on 2nd January. The companies that were considered were Intel, Bungie and SK Hynix. Intel has better technical support. Intel has very long waits for delivery and poor customer service. Unprofessional behavior exhibited by the Intel representative. SK Hynix is a proven supplier with established quality standards. Storage is easier to integrate with existing systems. Bungie has a quicker turnaround time and bigger budget. Bungie has great customer service. Bungie's representatives are very professional and courteous. The senior manager spoke of his experience of twenty years of working with Intel and that the downsides of working with Intel are a phenomenon of recent months and hence shouldn't be considered while making a decade-long plan.
Response"""
OUTPUT -
"Intel has better technical support. Intel has very long waits for delivery and poor customer service.\n\nSK Hynix is a proven supplier with established quality standards.\n\nBungie has a quicker turnaround time and bigger budget. Bungie has great customer service.\n\nThe senior manager spoke of his experience of twenty years of working with Intel and that the downsides of working with Intel are a phenomenon of recent months and hence shouldn't be considered while making a decade-long plan."
My expectation is that it only tells about Intel in the response.
Can you please guide me what more can I do without training to improve the responses?
Try this:
A discussion about sourcing part PN569 for the company's project took place on 2nd January. The companies that were considered were Intel, Bungie and SK Hynix. Intel has better technical support. Intel has very long waits for delivery and poor customer service. Unprofessional behavior exhibited by the Intel representative. SK Hynix is a proven supplier with established quality standards. Storage is easier to integrate with existing systems. Bungie has a quicker turnaround time and bigger budget. Bungie has great customer service. Bungie's representatives are very professional and courteous. The senior manager spoke of his experience of twenty years of working with Intel and that the downsides of working with Intel are a phenomenon of recent months and hence shouldn't be considered while making a decade-long plan.
Summarize the supplier Intel.
... yields ...
Intel is the preferred supplier for the project as they have the best technical support, better customer service and better turnaround time. They have a reputation of unprofessional behavior and their representatives are not professional and courteous.
Thank you, will do!
Follow up question: If I wish to finetune dolly 12b v2 for the segmentation/summarization task, how many training instances would I need? I understand the more the merrier but what's like a decent minimum number?
I suppose you could note that 15K examples was roughly enough to tune for instruction-following, and summarization is probably a relatively easier task, so, on the order of 10000? No real hard and fast rule there.
Thank you Sean! I was under the impression that summarization is one of the "instruction" that it is trained to follow. Can you please clarify how instruction following differs from summarization?
It is, it's not a summarization-only model, but one of the types of prompt-response pairs that were written were along the lines of "summarize X from this context Y"
Thank you