metadata

language:
  - en
license: cc-by-nc-4.0
tags:
  - mental
  - mental health
  - large language model
  - alpaca
model-index:
  - name: mental-alpaca
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 28.58
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 26.02
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 27.04
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 48.61
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 48.38
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 0
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
          name: Open LLM Leaderboard

Model Card for mental-alpaca

This is a fine-tuned large language model for mental health prediction via online text data.

Model Details

Model Description

We fine-tune an Alpaca model with 4 high-quality text (6 tasks in total) datasets for the mental health prediction scenario: Dreaddit, DepSeverity, SDCNL, and CCRS-Suicide. We have a separate model, fine-tuned on FLAN-T5-XXL, namely Mental-FLAN-T5, shared here

Developed by: Northeastern University Human-Centered AI Lab
Model type: Sequence-to-sequence Text-generation
Language(s) (NLP): English
License: cc-by-nc-4.0
Finetuned from model: https://github.com/tatsu-lab/stanford_alpaca

Model Sources [optional]

Repository: https://github.com/neuhai/Mental-LLM
Paper: https://arxiv.org/abs/2307.14385

Uses

Direct Use

The model is intended to be used for research purposes only in English. The model has been fine-tuned for mental health prediction via online text data. Detailed information about the fine-tuning process and prompts can be found in our paper. The use of this model should also comply with the restrictions from stanford_alpaca project and Llama-2-7b.

Out-of-Scope Use

The out-of-scope use of this model should comply with stanford_alpaca project and Llama-2-7b.

Bias, Risks, and Limitations

The Bias, Risks, and Limitations of this model should also comply with stanford_alpaca project and Llama-2-7b.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NEU-HAI/Llama-2-7b-alpaca-cleaned")
model = AutoModelForCausalLM.from_pretrained("NEU-HAI/Llama-2-7b-alpaca-cleaned")

Training Details and Evaluation

Detailed information about our work can be found in our paper.

Citation

@article{xu2023leveraging,
  title={Mental-LLM: Leveraging large language models for mental health prediction via online text data},
  author={Xu, Xuhai and Yao, Bingshen and Dong, Yuanzhe and Gabriel, Saadia and Yu, Hong and Ghassemi, Marzyeh and Hendler, James and Dey, Anind K and Wang, Dakuo},
  journal={arXiv preprint arXiv:2307.14385},
  year={2023}
}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	29.77
AI2 Reasoning Challenge (25-Shot)	28.58
HellaSwag (10-Shot)	26.02
MMLU (5-Shot)	27.04
TruthfulQA (0-shot)	48.61
Winogrande (5-shot)	48.38
GSM8k (5-shot)	0.00