metadata

inference: false

weblab-10b-instruction-sft-GPTQ

original model weblab-10b-instruction-sft

This is 4bit GPTQ Version.

The size is smaller and the execution speed is faster, but the inference performance may be a little worse.

sample code

At least one GPU is currently required due to a limitation of the Accelerate library.
So this model cannot be run with the huggingface space free version.

pip install auto-gptq

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

quantized_model_dir = "dahara1/weblab-10b-instruction-sft-GPTQ"
model_basename = "gptq_model-4bit-128g"

tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)

model = AutoGPTQForCausalLM.from_quantized(
        quantized_model_dir,
        model_basename=model_basename,
        use_safetensors=True,
        device="cuda:0")


prompt_text = "スタジオジブリの作品を５つ教えてください"
prompt_template = f'以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。\n\n### 指示:\n{prompt_text}\n\n### 応答:'

tokens = tokenizer(prompt_template, return_tensors="pt").to("cuda:0").input_ids
output = model.generate(input_ids=tokens, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0]))

Benchmark

The results below are preliminary. The blank part is under measurement.
Also, the score may change as a result of tuning after this.

Japanese benchmark

We used Stability-AI/lm-evaluation-harness + gptq patch for evaluation.
The 4-task average accuracy is based on results of JCommonsenseQA-1.1, JNLI-1.1, MARC-ja-1.1, and JSQuAD-1.1.
model loading is performed with gptq_use_triton=True, and evaluation is performed with template version 0.3 using the few-shot in-context learning.
The number of few-shots is 3,3,3,2.

Model	Average	JCommonsenseQA	JNLI	MARC-ja	JSQuAD
weblab-10b-instruction-sft	78.78	74.35	65.65	96.06	79.04
weblab-10b	66.38	65.86	54.19	84.49	60.98
weblab-10b-instruction-sft-GPTQ	-	74.53	41.70	-	72.69

dahara1
/

weblab-10b-instruction-sft-GPTQ

weblab-10b-instruction-sft-GPTQ

sample code

Other documents

Benchmark