--- library_name: peft license: llama3.3 base_model: meta-llama/Llama-3.3-70B-Instruct tags: - axolotl - generated_from_trainer datasets: - anthracite-org/c2_logs_32k_llama3_qwen2_v1.3 - anthracite-core/Gryphe-Opus-Charcard-Roleplay - anthracite-org/kalo-opus-instruct-22k-no-refusal - lodrick-the-lafted/kalo-opus-instruct-3k-filtered - anthracite-org/nopm_claude_writing_fixed - anthracite-org/kalo_opus_misc_240827 - anthracite-org/kalo_misc_part2 model-index: - name: magnum-v4-se-70b-lora results: [] language: - en pipeline_tag: text-generation --- # Magnum-v4-SE-70B-LoRA The Magnum v4 series is complete, but here's something a little extra I wanted to tack on as I wasn't entirely satisfied with the results of v4 72B. "SE" for Special Edition - this model is finetuned from [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) as an rsLoRA adapter. The dataset is a slightly revised variant of the v4 data with some elements of the v2 data re-introduced. The objective, as with the other Magnum models, is to emulate the prose style and quality of the Claude 3 Sonnet/Opus series of models on a local scale, so don't be surprised to see "Claude-isms" in its output. [Merged full model](https://huggingface.co/Doctor-Shotgun/L3.3-70B-Magnum-v4-SE) ## Intended uses and limitations This model is intended for creative writing and roleplay purposes. It may show biases similar to those observed in contemporary LLM-based roleplay, in addition to those exhibited by the Claude 3 series of models and the base model. All outputs should be considered fiction, as this model is not intended to provide factual information or advice. ## Training procedure [WandB](https://wandb.ai/doctorshotgun/70b-magnum-lora/runs/ccq5l1a7?nw=nwuserdoctorshotgun) [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.6.0` ```yaml base_model: meta-llama/Llama-3.3-70B-Instruct base_model_ignore_patterns: "*/*" # optionally might have model_type or tokenizer_type model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer # Automatically upload checkpoint and final model to HF hub_model_id: Doctor-Shotgun/magnum-v4-se-70b-lora hub_strategy: "all_checkpoints" push_dataset_to_hub: hf_use_auth_token: true load_in_8bit: false load_in_4bit: false strict: false datasets: - path: anthracite-org/c2_logs_32k_llama3_qwen2_v1.3 type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-core/Gryphe-Opus-Charcard-Roleplay type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/kalo-opus-instruct-22k-no-refusal type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: lodrick-the-lafted/kalo-opus-instruct-3k-filtered type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/nopm_claude_writing_fixed type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/kalo_opus_misc_240827 type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn - path: anthracite-org/kalo_misc_part2 type: chat_template chat_template: llama3 roles_to_train: ["gpt"] field_messages: conversations message_field_role: from message_field_content: value train_on_eos: turn shuffle_merged_datasets: true dataset_prepared_path: /home/docshotgun/data/magnum-70b-data val_set_size: 0.0 output_dir: /home/docshotgun/data/70b-lora-out plugins: - axolotl.integrations.liger.LigerPlugin liger_rope: true liger_rms_norm: true liger_layer_norm: true liger_glu_activation: true liger_fused_linear_cross_entropy: true sequence_len: 32768 sample_packing: true eval_sample_packing: false pad_to_sequence_len: true adapter: lora lora_model_dir: lora_r: 128 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: peft_use_rslora: true lora_modules_to_save: - embed_tokens - lm_head wandb_project: 70b-magnum-lora wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 1 micro_batch_size: 1 num_epochs: 2 optimizer: paged_ademamix_8bit lr_scheduler: cosine learning_rate: 4.0e-5 max_grad_norm: 3.0 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: unsloth early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true s2_attention: warmup_steps: 40 evals_per_epoch: eval_table_size: eval_max_new_tokens: saves_per_epoch: 2 debug: deepspeed: ./deepspeed_configs/zero3_bf16.json weight_decay: 0.01 fsdp: fsdp_config: special_tokens: pad_token: <|finetune_right_pad_id|> ```

### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 4e-05 - train_batch_size: 1 - eval_batch_size: 1 - seed: 42 - distributed_type: multi-GPU - num_devices: 8 - total_train_batch_size: 8 - total_eval_batch_size: 8 - optimizer: Use paged_ademamix_8bit and the args are: No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 40 - num_epochs: 2 ### Framework versions - PEFT 0.14.0 - Transformers 4.47.1 - Pytorch 2.5.1+cu124 - Datasets 3.2.0 - Tokenizers 0.21.0