nmarafo/Mistral-7B-Instruct-v0.2-TrueFalse-Feedback-GPTQ

Model Card for Model ID

This is an adapter prepared to return True or False depending on whether the student's answer ("student_answer") is correct based on the question ("question") and comparing it with a given answer ("best_answer"). The prompt has the following structure:

<s>[INST]Analyze the question, the expected answer, and the student's response.
Determine if the student's answer is correct or not. It only returns True if the student's answer is correct with respect to the expected answer or False otherwise.
Add a brief comment explaining why the answer is correct or incorrect.\n\n
Question: {question}\n
Expected Answer: {best_answer}\n
Student Answer: {student_answer}[/INST]"

How to Get Started with the Model

In Google Colab:

!pip install -q -U transformers peft accelerate optimum
!pip install datasets==2.15.0
!pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu117/

from peft import AutoPeftModelForCausalLM
from rich import print
from transformers import GenerationConfig, AutoTokenizer

import torch

model_id = "TheBloke/Mistral-7B-Instruct-v0.2-GPTQ"
adapter = "nmarafo/Mistral-7B-Instruct-v0.2-TrueFalse-Feedback-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True, return_token_type_ids=False)
tokenizer.pad_token = tokenizer.eos_token

model = AutoPeftModelForCausalLM.from_pretrained(adapter, low_cpu_mem_usage=True, return_dict=True, torch_dtype=torch.float16, device_map="cuda")

def predict(question, best_answer, student_answer):
    system_message = "Analyze the question, the expected answer, and the student's response. Determine if the student's answer is conceptually correct in relation to the expected answer, regardless of the exact wording. Return True if the student's answer is correct or False otherwise. Add a brief comment explaining the rationale behind the answer being correct or incorrect."
    prompt = f"{system_message}\n\nQuestion: {question}\nBest Answer: {best_answer}\nStudent Answer: {student_answer}"
    prompt_template=f"<s>[INST]{prompt}[/INST]"

    encoding = tokenizer(prompt_template, return_tensors='pt', padding=True, truncation=True, max_length=512)
    input_ids = encoding['input_ids'].cuda()
    attention_mask = encoding['attention_mask'].cuda()

    output = model.generate(input_ids, attention_mask=attention_mask, 
                            temperature=0.7, do_sample=True, top_p=0.95, 
                            top_k=40, max_new_tokens=512, pad_token_id=tokenizer.eos_token_id)
    response = tokenizer.decode(output[0], skip_special_tokens=True)
    return response

question="Mention all the Canary Island"
best_answer="Tenerife, Fuerteventura, Gran Canaria, Lanzarote, La Palma, La Gomera, El Hierro, La Graciosa"
student_answer="Tenerife"

print(predict(question, best_answer, student_answer))    

To perform inference on the test dataset example load the model from the checkpoint

persisted_model = AutoPeftModelForCausalLM.from_pretrained( adapter, low_cpu_mem_usage=True, return_dict=True, torch_dtype=torch.float16, device_map="cuda")

Some gen config knobs

generation_config = GenerationConfig( penalty_alpha=0.6, do_sample = True, top_k=5, temperature=0.5, repetition_penalty=1.2, max_new_tokens=512 )

Framework versions

• PEFT 0.8.2