---
language:
- en
license: cc-by-nc-4.0
tags:
- mental
- mental health
- large language model
- alpaca
model-index:
- name: mental-alpaca
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 28.58
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 26.02
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 27.04
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 48.61
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 48.38
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 0.0
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=NEU-HAI/mental-alpaca
      name: Open LLM Leaderboard
---
# Model Card for mental-alpaca

<!-- Provide a quick summary of what the model is/does. -->

This is a fine-tuned large language model for mental health prediction via online text data.

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

We fine-tune an Alpaca model with 4 high-quality text (6 tasks in total) datasets for the mental health prediction scenario: Dreaddit, DepSeverity, SDCNL, and CCRS-Suicide.
We have a separate model, fine-tuned on FLAN-T5-XXL, namely Mental-FLAN-T5, shared [here](https://huggingface.co/NEU-HAI/mental-flan-t5-xxl) 


- **Developed by:** Northeastern University Human-Centered AI Lab
- **Model type:** Sequence-to-sequence Text-generation
- **Language(s) (NLP):** English
- **License:** cc-by-nc-4.0 
- **Finetuned from model:** https://github.com/tatsu-lab/stanford_alpaca

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/neuhai/Mental-LLM
- **Paper:** https://arxiv.org/abs/2307.14385

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

The model is intended to be used for research purposes only in English.
The model has been fine-tuned for mental health prediction via online text data. Detailed information about the fine-tuning process and prompts can be found in our [paper](https://arxiv.org/abs/2307.14385).
The use of this model should also comply with the restrictions from [stanford_alpaca project](https://github.com/tatsu-lab/stanford_alpaca) and [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b).



### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

The out-of-scope use of this model should comply with [stanford_alpaca project](https://github.com/tatsu-lab/stanford_alpaca) and [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b).


## Bias, Risks, and Limitations

The Bias, Risks, and Limitations of this model should also comply with [stanford_alpaca project](https://github.com/tatsu-lab/stanford_alpaca) and [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b).


## How to Get Started with the Model

Use the code below to get started with the model.

```
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NEU-HAI/Llama-2-7b-alpaca-cleaned")
model = AutoModelForCausalLM.from_pretrained("NEU-HAI/Llama-2-7b-alpaca-cleaned")
```

## Training Details and Evaluation


Detailed information about our work can be found in our [paper](https://arxiv.org/abs/2307.14385).

## Citation

```
@article{xu2023leveraging,
  title={Mental-LLM: Leveraging large language models for mental health prediction via online text data},
  author={Xu, Xuhai and Yao, Bingshen and Dong, Yuanzhe and Gabriel, Saadia and Yu, Hong and Ghassemi, Marzyeh and Hendler, James and Dey, Anind K and Wang, Dakuo},
  journal={arXiv preprint arXiv:2307.14385},
  year={2023}
}
```

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_NEU-HAI__mental-alpaca)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |29.77|
|AI2 Reasoning Challenge (25-Shot)|28.58|
|HellaSwag (10-Shot)              |26.02|
|MMLU (5-Shot)                    |27.04|
|TruthfulQA (0-shot)              |48.61|
|Winogrande (5-shot)              |48.38|
|GSM8k (5-shot)                   | 0.00|