TW-Pythia-6.9B-Chat
Taiwanese Mandarin Pythia Language Model, instruction-tuned for dialogue.
Version 0.2
Model Details
The TW-Pythia model is derived from the Apache-2.0-licenced Pythia language model, with 8000 new Traditional Chinese tokens added, embed layers resized and re-trained.
Basics
- Developed by: @zetavg based on EleutherAI's Pythia language model.
- Model type: Transformer-based GPT-NeoX Causal Language Model
- Languages: English, Traditional Chinese
- License: Unknown due to unconfirmed usage license of the training data
- Derived from model: EleutherAI/pythia-6.9b
Model Sources
- Repository: https://github.com/zetavg/twlm
- Demo: See https://hackmd.io/@z/twlm-demo
Uses
Currently, this model has not demonstrated any practical value in Traditional Chinese processing without further training, but it does possess some basic Chinese-English translation capabilities.
Training Details
Training Data
- 200k English ↔ Traditional Chinese Sentences from the COCT Database.
- ~8k English and Traditional Chinese mixed ShareGPT data.
Training Procedure
First, we build a BPE tokenizer based on the original Pythia tokenizer with 8000 new Traditional Chinese tokens added.
Then, we resize the embedding layer of the pythia-6.9b model to accommodate the new vocabulary size, and we train only the input/output embedding layers to allow the model to learn the new Traditional Chinese words and phrases.
At last, LoRA weights are added to the model and fine-tuned for instruction following.
Training Hyperparameters
- Training regime:
fp32 - See: https://github.com/zetavg/twlm/blob/main/configs/ta01_p7b.yaml
Hardware
- 1xH100 80GB GPU on Lambda Cloud (with Skypilot), about 20h in total.
- Downloads last month
- 14