fix: update tokenizer config to support `add_generation_prompt=True` and clarify content

#57

by lamhieu - opened May 11

base: refs/heads/main

←

from: refs/pr/57

Discussion Files changed

-1

lamhieu

May 11

Here I have updated it so that for each role it will return the correct tokens id of that role more clearly, trim the spaces and support add_generation_prompt config.

fix: update tokenizer config to support `add_generation_prompt=True` and clarify contenta5f0652d

nguyenbh changed pull request status to closed Jul 1

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment