zephyr-7b-gemma-sft-african-ultrachat-5k

This model is a fine-tuned version of google/gemma-7b on the masakhane/african-ultrachat and the israel/untrachat_en datasets. It achieves the following results on the evaluation set:

  • Loss: 1.1356

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 16
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss
1.1994 1.0 2480 1.1954
1.0039 2.0 4960 1.0974
0.6836 3.0 7440 1.1356

Framework versions

  • Transformers 4.39.0.dev0
  • Pytorch 2.2.1+cu121
  • Datasets 2.14.6
  • Tokenizers 0.15.2

Usage


# Install transformers from source - only needed for versions <= v4.34
# pip install git+https://github.com/huggingface/transformers.git
# pip install accelerate

import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="
zephyr-7b-gemma-sft-african-ultrachat-5k", torch_dtype=torch.bfloat16, device_map="auto")

# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who answewrs question in given language",
    },
    {"role": "user", "content": "what is the 3 biggest countrys in Africa?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
# <|system|>
# You are a friendly chatbot who always responds in the style of a pirate<eos>
# <|user|>
# what is the 3 biggest countrys in Africa?<eos>
# <|assistant|>
# The 3 biggest countries in Africa are Nigeria, Ethiopia and South Africa.

Quantized Versions through bitsandbytes


import torch
from transformers import pipeline
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig


quantization_config = BitsAndBytesConfig(load_in_4bit=True)

tokenizer = AutoTokenizer.from_pretrained("
zephyr-7b-gemma-sft-african-ultrachat-5k")
model = AutoModelForCausalLM.from_pretrained("
zephyr-7b-gemma-sft-african-ultrachat-5k", quantization_config=quantization_config)


pipe = pipeline("text-generation", model=model,tokenizer=tokenizer, torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who answewrs question in given language",
    },
    {"role": "user", "content": "list languages in Africa?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
Downloads last month
31
Safetensors
Model size
8.54B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for masakhane/zephyr-7b-gemma-sft-african-ultrachat-5k

Base model

google/gemma-7b
Finetuned
(97)
this model

Datasets used to train masakhane/zephyr-7b-gemma-sft-african-ultrachat-5k