---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
inference: false
datasets:
- the_pile_books3
tags:
- mosaicML
- sharded
- story
---

# mpt-7b-storywriter: sharded


<a href="https://colab.research.google.com/gist/pszemraj/a979cdcc02edb916661c5dd97cf2294e/mpt-storywriter-sharded-inference.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

This is a version of the [mpt-7b-storywriter](https://huggingface.co/mosaicml/mpt-7b-storywriter)  model, sharded to 2 GB chunks for low-RAM loading (i.e. Colab).  The weights are stored in `bfloat16` so in theory you can run this on CPU, though it may take forever.

Please refer to the previously linked repo for details on usage/implementation/etc. This model was downloaded from the original repo under Apache-2.0 and is redistributed under the same license. 


## Basic Usage

> Note when using: this is **not** an instruction-tuned model, so you need to give it sufficient input text to continue generating something on-topic with your prompt
> 
Install/upgrade packages:

```bash
pip install -U torch transformers accelerate einops
```

Load the model:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = 'ethzanalytics/mpt-7b-storywriter-sharded'
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    revision='197d14245ad874da82194248cab1ce8cf87fa713', # optional, but a good idea
    device_map='auto',
    load_in_8bit=False, # install bitsandbytes then set to true for 8-bit
)
model = torch.compile(model)
tokenizer = AutoTokenizer.from_pretrained(model_name)
```

Then you can use `model.generate()` as you would normally - see the notebook for details.


---