--- library_name: peft license: gemma base_model: google/gemma-2-27b-it tags: - axolotl - generated_from_trainer datasets: - databricks/databricks-dolly-15k model-index: - name: gemma-2-27b-it-dolly-15k results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.6.0` ```yaml # base_model: meta-llama/Llama-3.2-1B-Instruct # # Automatically upload checkpoint and final model to HF # # hub_model_id: kweinmeister/Llama-3.2-1B-Instruct-MetaMathQA # hub_model_id: kweinmeister/Llama-3.2-1B-Instruct-gsm8k # load_in_8bit: false # load_in_4bit: true # strict: false # datasets: # - path: openai/gsm8k # type: alpaca_chat.load_qa # name: "main" # train_on_split: "train" # # datasets: # # - path: meta-math/MetaMathQA # # type: # # field_instruction: query # # field_output: response # val_set_size: 0.1 # # output_dir: "/mnt/disks/gcs/axolotl/outputs/out" # output_dir: "/mnt/disks/gcs/axolotl/outputs/gsm8k-out" # # output_dir: "/mnt/disks/gcs/axolotl/outputs/MetaMathQA-out" # adapter: qlora # lora_model_dir: # sequence_len: 2048 # sample_packing: true # eval_sample_packing: true # pad_to_sequence_len: true # lora_r: 32 # lora_alpha: 16 # lora_dropout: 0.05 # lora_fan_in_fan_out: # lora_target_modules: # - gate_proj # - down_proj # - up_proj # - q_proj # - v_proj # - k_proj # - o_proj # wandb_project: # wandb_entity: # wandb_watch: # wandb_name: # wandb_log_model: # gradient_accumulation_steps: 4 # micro_batch_size: 2 # num_epochs: 3 # # optimizer: adamw_bnb_8bit # optimizer: adamw_torch # lr_scheduler: cosine # learning_rate: 2e-5 # train_on_inputs: false # group_by_length: false # bf16: auto # fp16: # tf32: false # # gradient_checkpointing: true # gradient_checkpointing: false # early_stopping_patience: # resume_from_checkpoint: # local_rank: # logging_steps: 1 # xformers_attention: # flash_attention: true # loss_watchdog_threshold: 5.0 # loss_watchdog_patience: 3 # warmup_steps: 10 # evals_per_epoch: 4 # eval_table_size: # eval_max_new_tokens: 128 # saves_per_epoch: 1 # debug: # deepspeed: # weight_decay: 0.0 # # fsdp: # # fsdp_config: # fsdp: # - full_shard # - auto_wrap # fsdp_config: # fsdp_limit_all_gathers: true # fsdp_sync_module_states: true # fsdp_offload_params: true # fsdp_use_orig_params: false # fsdp_cpu_ram_efficient_loading: true # fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP # fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer # fsdp_state_dict_type: FULL_STATE_DICT # fsdp_sharding_strategy: FULL_SHARD # fsdp_activation_checkpointing: true # special_tokens: # # pad_token: "<|end_of_text|>" # special_tokens: # bos_token: "<|begin_of_text|>" # eos_token: "<|eot_id|>" # pad_token: "<|finetune_right_pad_id|>" base_model: google/gemma-2-27b-it # model_type: AutoModelForCausalLM # tokenizer_type: AutoTokenizer hub_model_id: kweinmeister/gemma-2-27b-it-dolly-15k load_in_8bit: false load_in_4bit: true strict: false datasets: - path: databricks/databricks-dolly-15k type: field_instruction: instruction field_input: context field_output: response val_set_size: 0.1 output_dir: "/mnt/disks/gcs/axolotl/outputs/dolly-15k-out" adapter: qlora lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true sequence_len: 2048 sample_packing: true # eval_sample_packing: true pad_to_sequence_len: true gradient_accumulation_steps: 4 micro_batch_size: 2 num_epochs: 3 # optimizer: adamw_bnb_8bit optimizer: adamw_torch lr_scheduler: cosine learning_rate: 2e-5 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false # gradient_checkpointing: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: false # loss_watchdog_threshold: 5.0 # loss_watchdog_patience: 3 warmup_ratio: 0.1 evals_per_epoch: 4 eval_max_new_tokens: 128 saves_per_epoch: 1 debug: # deepspeed: weight_decay: 0.0 deepspeed: deepspeed_configs/zero1.json fsdp: fsdp_config: # fsdp: # - full_shard # - auto_wrap # fsdp_config: # fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP # fsdp_backward_prefetch: BACKWARD_PRE # fsdp_cpu_ram_efficient_loading: true # fsdp_forward_prefetch: false # fsdp_offload_params: true # fsdp_sharding_strategy: FULL_SHARD # fsdp_state_dict_type: SHARDED_STATE_DICT # fsdp_transformer_layer_cls_to_wrap: GemmaDecoderLayer # fsdp_sync_module_states: true # fsdp_use_orig_params: true # fsdp_config: # fsdp_limit_all_gathers: true # fsdp_sync_module_states: true # fsdp_offload_params: true # fsdp_use_orig_params: false # fsdp_cpu_ram_efficient_loading: true # fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP # fsdp_transformer_layer_cls_to_wrap: GemmaDecoderLayer # fsdp_state_dict_type: FULL_STATE_DICT # fsdp_sharding_strategy: FULL_SHARD # fsdp_activation_checkpointing: true # special_tokens: # # pad_token: "<|end_of_text|>" # special_tokens: # bos_token: "<|begin_of_text|>" # eos_token: "<|eot_id|>" # pad_token: "<|finetune_right_pad_id|>" ```

# gemma-2-27b-it-dolly-15k This model is a fine-tuned version of [google/gemma-2-27b-it](https://huggingface.co/google/gemma-2-27b-it) on the databricks/databricks-dolly-15k dataset. It achieves the following results on the evaluation set: - Loss: 1.6809 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - num_devices: 2 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - total_eval_batch_size: 4 - optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 23 - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 3.8741 | 0.0129 | 1 | 4.1287 | | 3.5275 | 0.2589 | 20 | 3.7627 | | 2.5496 | 0.5178 | 40 | 2.5361 | | 2.1047 | 0.7767 | 60 | 2.0215 | | 1.8435 | 1.0259 | 80 | 1.8475 | | 1.8821 | 1.2848 | 100 | 1.7748 | | 1.834 | 1.5437 | 120 | 1.7345 | | 1.7633 | 1.8026 | 140 | 1.7098 | | 1.6382 | 2.0647 | 160 | 1.6954 | | 1.9356 | 2.3236 | 180 | 1.6863 | | 1.6196 | 2.5825 | 200 | 1.6819 | | 1.7489 | 2.8414 | 220 | 1.6809 | ### Framework versions - PEFT 0.14.0 - Transformers 4.47.1 - Pytorch 2.3.1+cu121 - Datasets 3.1.0 - Tokenizers 0.21.0