| # pulze-intent-v0.1 | |
| Intent-tuned LLM router that selects the best LLM for a user query. Use with [knn-router](https://github.com/pulzeai-oss/knn-router). | |
| ## Models | |
| - claude-3-haiku-20240307 | |
| - claude-3-opus-20240229 | |
| - claude-3-sonnet-20240229 | |
| - command-r | |
| - command-r-plus | |
| - dbrx-instruct | |
| - gpt-3.5-turbo-0125 | |
| - gpt-4-turbo-2024-04-09 | |
| - llama-3-70b-instruct | |
| - mistral-large | |
| - mistral-medium | |
| - mistral-small | |
| - mixtral-8x7b-instruct | |
| ## Data | |
| ### Prompts and Intent Categories | |
| Prompt and intent categories are derived from the [GAIR-NLP/Auto-J scenario classification dataset](https://github.com/GAIR-NLP/auto-j/blob/2ae17a3965d933232e9cd50302aa0f176249c83b/README.md?plain=1#L582). | |
| Citation: | |
| ``` | |
| @article{li2023generative, | |
| title={Generative Judge for Evaluating Alignment}, | |
| author={Li, Junlong and Sun, Shichao and Yuan, Weizhe and Fan, Run-Ze and Zhao, Hai and Liu, Pengfei}, | |
| journal={arXiv preprint arXiv:2310.05470}, | |
| year={2023} | |
| } | |
| ``` | |
| ### Response Evaluation | |
| Candidate model responses were evaluated pairwise using `openai/gpt-4-turbo-2024-04-09`, with the following prompt: | |
| ``` | |
| You are an expert, impartial judge tasked with evaluating the quality of responses generated by two AI assistants. | |
| Think step by step, and evaluate the responses, <response1> and <response2> to the instruction, <instruction>. Follow these guidelines: | |
| - Avoid any position bias and ensure that the order in which the responses were presented does not influence your judgement | |
| - Do not allow the length of the responses to influence your judgement - a concise response can be as effective as a longer one | |
| - Consider factors such as adherence to the given instruction, helpfulness, relevance, accuracy, depth, creativity, and level of detail | |
| - Be as objective as possible | |
| Make your decision on which of the two responses is better for the given instruction from the following choices: | |
| If <response1> is better, use "1". | |
| If <response2> is better, use "2". | |
| If both answers are equally good, use "0". | |
| If both answers are equally bad, use "0". | |
| <instruction> | |
| {INSTRUCTION} | |
| </instruction> | |
| <response1> | |
| {RESPONSE1} | |
| </response1> | |
| <response2> | |
| {RESPONSE2} | |
| </response2> | |
| ``` | |
| Each pair of models is subject to 2 matches, with the positions of the respective responses swapped in the evaluation prompt. A model is considered a winner only if | |
| it wins both matches. | |
| For each prompt, we then compute Bradley-Terry scores for the respective models using the same [method](https://github.com/lm-sys/FastChat/blob/f2e6ca964af7ad0585cadcf16ab98e57297e2133/fastchat/serve/monitor/elo_analysis.py#L57) as that used in the [LMSYS Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). Finally, we normalize all scores to a scale from 0 to 1 for interoperability with other weighted ranking systems. | |
| ## Model | |
| The embedding model was generated by first fine-tuning [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) with the intent categories from the dataset above, using contrastive learning with cosine similarity loss, and subsequently merging the resultant model with the base model at a 3:2 ratio. |