Model Card for vantaa32/llama-2-7b-fourierft-alpaca

meta-llama/Llama-2-7b-hf fine-tuned on alpaca-cleaned dataset using FourierFT

Fine-tuned with hyperparameters specified in the paper + weight decay of 2e-3

Model Details

Model Description

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

[More Information Needed]

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

[More Information Needed]

Bias, Risks, and Limitations

[More Information Needed]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

Training Details

Training Data

[More Information Needed]

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime: [More Information Needed]

export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1
python SFT.py \
    --model_tag llama-2-7b \
    --model_name_or_path meta-llama/Llama-2-7b-hf \
    --n_frequency 1000 \
    --num_train_epochs 1 \
    --weight_decay 2e-3 \
    --learning_rate 0.03 \
    --scale 300.0 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --warmup_ratio 0.06 \
    --output_dir output-peft-lib

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

MT-Bench 3.789308

MMLU: 0.457

Average accuracy 0.320 - abstract_algebra
Average accuracy 0.444 - anatomy
Token indices sequence length is longer than the specified maximum sequence length for this model (658 > 512). Running this sequence through the model will result in indexing errors
Average accuracy 0.414 - astronomy
Average accuracy 0.500 - business_ethics
Average accuracy 0.464 - clinical_knowledge
Average accuracy 0.451 - college_biology
Average accuracy 0.320 - college_chemistry
Average accuracy 0.340 - college_computer_science
Average accuracy 0.360 - college_mathematics
Average accuracy 0.445 - college_medicine
Average accuracy 0.186 - college_physics
Average accuracy 0.550 - computer_security
Average accuracy 0.409 - conceptual_physics
Average accuracy 0.289 - econometrics
Average accuracy 0.441 - electrical_engineering
Average accuracy 0.286 - elementary_mathematics
Average accuracy 0.310 - formal_logic
Average accuracy 0.320 - global_facts
Average accuracy 0.484 - high_school_biology
Average accuracy 0.330 - high_school_chemistry
Average accuracy 0.420 - high_school_computer_science
Average accuracy 0.600 - high_school_european_history
Average accuracy 0.530 - high_school_geography
Average accuracy 0.674 - high_school_government_and_politics
Average accuracy 0.449 - high_school_macroeconomics
Average accuracy 0.289 - high_school_mathematics
Average accuracy 0.420 - high_school_microeconomics
Average accuracy 0.325 - high_school_physics
Average accuracy 0.631 - high_school_psychology
Average accuracy 0.287 - high_school_statistics
Average accuracy 0.583 - high_school_us_history
Average accuracy 0.637 - high_school_world_history
Average accuracy 0.574 - human_aging
Average accuracy 0.565 - human_sexuality
Average accuracy 0.636 - international_law
Average accuracy 0.509 - jurisprudence
Average accuracy 0.564 - logical_fallacies
Average accuracy 0.384 - machine_learning
Average accuracy 0.553 - management
Average accuracy 0.701 - marketing
Average accuracy 0.540 - medical_genetics
Average accuracy 0.641 - miscellaneous
Average accuracy 0.512 - moral_disputes
Average accuracy 0.238 - moral_scenarios
Average accuracy 0.500 - nutrition
Average accuracy 0.582 - philosophy
Average accuracy 0.503 - prehistory
Average accuracy 0.358 - professional_accounting
Average accuracy 0.351 - professional_law
Average accuracy 0.540 - professional_medicine
Average accuracy 0.433 - professional_psychology
Average accuracy 0.536 - public_relations
Average accuracy 0.429 - security_studies
Average accuracy 0.602 - sociology
Average accuracy 0.650 - us_foreign_policy
Average accuracy 0.428 - virology
Average accuracy 0.690 - world_religions
Average accuracy 0.297 - math
Average accuracy 0.496 - health
Average accuracy 0.355 - physics
Average accuracy 0.620 - business
Average accuracy 0.474 - biology
Average accuracy 0.327 - chemistry
Average accuracy 0.422 - computer science
Average accuracy 0.415 - economics
Average accuracy 0.441 - engineering
Average accuracy 0.408 - philosophy
Average accuracy 0.545 - other
Average accuracy 0.572 - history
Average accuracy 0.530 - geography
Average accuracy 0.554 - politics
Average accuracy 0.526 - psychology
Average accuracy 0.587 - culture
Average accuracy 0.380 - law
Average accuracy 0.363 - STEM
Average accuracy 0.430 - humanities
Average accuracy 0.512 - social sciences
Average accuracy 0.530 - other (business, health, misc.)
Average accuracy: 0.457

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Factors

[More Information Needed]

Metrics

[More Information Needed]

Results

[More Information Needed]

Summary

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Framework versions

• PEFT 0.14.0