--- license: apache-2.0 library_name: peft tags: - axolotl - generated_from_trainer base_model: NousResearch/Yarn-Mistral-7b-64k model-index: - name: taopanda-2_0a956f4b-2530-421e-9d67-9975bdd289dc results: [] --- [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.4.1` ```yaml adapter: lora base_model: NousResearch/Yarn-Mistral-7b-64k bf16: auto dataset_prepared_path: last_run_prepared datasets: - data_files: - 7708f8a044b2986d_train_data.json ds_type: json format: custom path: 7708f8a044b2986d_train_data.json type: field: null field_input: null field_instruction: prompt field_output: chosen field_system: null format: null no_input_format: null system_format: '{system}' system_prompt: '' debug: null deepspeed: null early_stopping_patience: null eval_max_new_tokens: 128 eval_table_size: null evals_per_epoch: 4 flash_attention: true fp16: null fsdp: null fsdp_config: null gradient_accumulation_steps: 4 gradient_checkpointing: true group_by_length: false hub_model_id: FatCat87/taopanda-2_0a956f4b-2530-421e-9d67-9975bdd289dc learning_rate: 0.0002 load_in_4bit: false load_in_8bit: true local_rank: null logging_steps: 1 lora_alpha: 16 lora_dropout: 0.05 lora_fan_in_fan_out: null lora_model_dir: null lora_r: 32 lora_target_linear: true lora_target_modules: - gate_proj - down_proj - up_proj - q_proj - v_proj - k_proj - o_proj loss_watchdog_patience: 3 loss_watchdog_threshold: 5.0 lr_scheduler: cosine micro_batch_size: 2 model_type: MistralForCausalLM num_epochs: 1 optimizer: adamw_bnb_8bit output_dir: ./outputs/lora-out/taopanda-2_0a956f4b-2530-421e-9d67-9975bdd289dc pad_to_sequence_len: true resume_from_checkpoint: null sample_packing: true saves_per_epoch: 1 seed: 29496 sequence_len: 8192 special_tokens: pad_token: strict: false tf32: false tokenizer_type: LlamaTokenizer train_on_inputs: false val_set_size: 0.1 wandb_entity: fatcat87-taopanda wandb_log_model: null wandb_mode: online wandb_name: taopanda-2_0a956f4b-2530-421e-9d67-9975bdd289dc wandb_project: subnet56 wandb_runid: taopanda-2_0a956f4b-2530-421e-9d67-9975bdd289dc wandb_watch: null warmup_steps: 10 weight_decay: 0.0 xformers_attention: null ```

[Visualize in Weights & Biases](https://wandb.ai/fatcat87-taopanda/subnet56/runs/eica4c2z) # taopanda-2_0a956f4b-2530-421e-9d67-9975bdd289dc This model is a fine-tuned version of [NousResearch/Yarn-Mistral-7b-64k](https://huggingface.co/NousResearch/Yarn-Mistral-7b-64k) on the None dataset. It achieves the following results on the evaluation set: - Loss: 1.3950 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 2 - eval_batch_size: 2 - seed: 29496 - distributed_type: multi-GPU - num_devices: 2 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - total_eval_batch_size: 4 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 2.019 | 0.0208 | 1 | 2.0295 | | 1.5424 | 0.25 | 12 | 1.5162 | | 1.4308 | 0.5 | 24 | 1.4270 | | 1.3993 | 0.75 | 36 | 1.3999 | | 1.3966 | 1.0 | 48 | 1.3950 | ### Framework versions - PEFT 0.11.1 - Transformers 4.42.3 - Pytorch 2.3.0+cu121 - Datasets 2.19.1 - Tokenizers 0.19.1