chat-prompt-detailed

README.md

@@ -31,7 +31,7 @@ Install `transformers`
 pip install transformers accelerate
 ```
 
-**Chat use:** The 70B Instruct model uses a different prompt template than the smaller versions. To use it with `transformers`, we recommend you use the built-in chat template:
+**Chat use:** The 70B Instruct model uses a [different prompt template](#chat_prompt) than the smaller versions. To use it with `transformers`, we recommend you use the built-in chat template:
 
  ```py
 from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -86,6 +86,69 @@ for seq in sequences:
     print(f"Result: {seq['generated_text']}")
 ```
 
+<a name="chat_prompt"></a>
+## Chat prompt
+
+CodeLlama 70B Instruct uses a different format for the chat prompt than previous Llama 2 or CodeLlama models. As mentioned above, the easiest way to use it is with the help of the tokenizer's chat template. If you need to build the string or tokens, manually, here's how to do it.
+
+We'll do our tests with the following made-up dialog:
+
+```py
+chat = [
+    {"role": "system", "content": "System prompt    "},
+    {"role": "user", "content": "First user query"},
+    {"role": "assistant", "content": "Model response to first query"},
+    {"role": "user", "content": "Second user query"},
+]
+```
+
+First, let's see what the prompt looks like if we use the chat template:
+
+```py
+tokenizer.apply_chat_template(chat, tokenize=False)
+```
+
+```
+'<s>Source: system\n\n System prompt <step> Source: user\n\n First user query <step> Source: assistant\n\n Model response to first query <step> Source: user\n\n Second user query <step> Source: assistant\nDestination: user\n\n '
+```
+
+So each turn of the conversation has a `Source` (`system`, `user`, or `assistant`), and then the content appears after two newlines and a space. Turns are separated with the special token ` <step> `. After the last turn (which must necessarily come from the `user`), we invite the model to respond by using the special syntax `Source: assistant\nDestination: user\n\n `. Let's see how we can build the same string ourselves:
+
+```py
+output = "<s>"
+for m in chat:
+    output += f"Source: {m['role']}\n\n {m['content'].strip()}"
+    output += " <step> "
+output += "Source: assistant\nDestination: user\n\n "
+output
+```
+
+```
+'<s>Source: system\n\n System prompt <step> Source: user\n\n First user query <step> Source: assistant\n\n Model response to first query <step> Source: user\n\n Second user query <step> Source: assistant\nDestination: user\n\n '
+```
+
+To verify that we got it right, we'll compare against the [reference code in the original GitHub repo](https://github.com/facebookresearch/codellama/blob/1af62e1f43db1fa5140fa43cb828465a603a48f3/llama/generation.py#L506). We used the same dialog and tokenized it with the `dialog_prompt_tokens` function and got the following tokens:
+
+```py
+reference_tokens = [1, 7562, 29901, 1788, 13, 13, 2184, 9508, 32015, 7562, 29901, 1404, 13, 13, 3824, 1404, 2346, 32015, 7562, 29901, 20255, 13, 13, 8125, 2933, 304, 937, 2346, 32015, 7562, 29901, 1404, 13, 13, 6440, 1404, 2346, 32015, 7562, 29901, 20255, 13, 14994, 3381, 29901, 1404, 13, 13, 29871]
+```
+
+Let's see what we get with the string we built using our Python loop. Note that we don't add "special tokens" because the string already starts with `<s>`, the beginning of sentence token:
+
+```py
+tokens = tokenizer.encode(output, add_special_tokens=False)
+assert reference_tokens == tokens
+```
+
+Similarly, let's verify that the chat template produces the same token sequence:
+
+```py
+assert reference_tokens == tokenizer.apply_chat_template(chat)
+```
+
+As a final detail, please note that if the dialog does not start with a `system` turn, the [original code will insert one with an empty content string](https://github.com/facebookresearch/codellama/blob/1af62e1f43db1fa5140fa43cb828465a603a48f3/llama/generation.py#L418).
+
+
 ## Model Details
 *Note: Use of this model is governed by the Meta license. Meta developed and publicly released the Code Llama family of large language models (LLMs).