---
license: cc-by-nc-4.0
datasets:
- tatsu-lab/alpaca
library_name: transformers
pipeline_tag: text-generation
tags:
- galactica
- alpaca
- opt
- gptq
inference: false
---
<!-- header start -->
<div style="width: 100%;">
    <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
</div>
<div style="display: flex; justify-content: space-between; width: 100%;">
    <div style="display: flex; flex-direction: column; align-items: flex-start;">
        <p><a href="https://discord.gg/Jq4vkcDakD">Chat & support: my new Discord server</a></p>
    </div>
    <div style="display: flex; flex-direction: column; align-items: flex-end;">
        <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
    </div>
</div>
<!-- header end -->

## About this repository
This is an attempt to create a GPTQ 4-bit version of [Galpaca 30B](https://huggingface.co/GeorgiaTechResearchInstitute/galpaca-30b).

I created these files on request. I have no previous experience of Galactica or Galpaca, and have not done much testing to confirm that the output is useful and usable.

You will need 18+ GB VRAM to load these models on a GPU.

## Provided files
* `galpaca-30B-4bit-128g.no-act-order.pt`
  * Created with: `python3 opt.py /content/galpaca-30b c4 --wbits 4 --new-eval --groupsize 128 --save galpaca-30B-4bit-128g.no-act-order.pt`
  * This file seems to create usable results, tested with [text-generation-webui](https://github.com/oobabooga/text-generation-webui).

* `galpaca-30B-4bit-128g.pt`
  * Created with: `python3 opt.py /content/galpaca-30b c4 --wbits 4 --new-eval --act-order --groupsize 128 --save galpaca-30B-4bit-128g.pt`
  * In my testing so far, **this file does not work**. It produces garbage output.
  * If you can get it working, please let me know!


## GPTQ

The GPTQ code used to create these models can be found at [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).

Note that during the GPTQ process, the following warning was seen:

`Token indices sequence length is longer than the specified maximum sequence length for this model (1915 > 512). Running this sequence through the model will result in indexing errors`

I do not know if this is indicates a potential problem in the GPTQ output, or if it can be ignored. If you know more about this, do let me know.


General model info below is as per the original [Galpaca 30B repository](https://huggingface.co/GeorgiaTechResearchInstitute/galpaca-30b).

For more information, example prompts and more, please see the original repository.

# GALPACA 30B (large)

GALACTICA 30B fine-tuned on the Alpaca dataset.

The model card from the original Galactica repo can be found [here](https://github.com/paperswithcode/galai/blob/main/docs/model_card.md), and the original paper [here](https://galactica.org/paper.pdf).

The dataset card for Alpaca can be found [here](https://huggingface.co/datasets/tatsu-lab/alpaca/blob/main/README.md), and the project homepage [here](https://crfm.stanford.edu/2023/03/13/alpaca.html).
  The Alpaca dataset was collected with a modified version of the [Self-Instruct Framework](https://github.com/yizhongw/self-instruct), and was built using OpenAI's `text-davinci-003` model. As such it is subject to OpenAI's terms of service.

<!-- footer start -->
## Discord

For further support, and discussions on these models and AI in general, join us at:

[TheBloke AI's Discord server](https://discord.gg/Jq4vkcDakD)

## Thanks, and how to contribute.

Thanks to the [chirper.ai](https://chirper.ai) team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.

* Patreon: https://patreon.com/TheBlokeAI
* Ko-Fi: https://ko-fi.com/TheBlokeAI

**Patreon special mentions**: Aemon Algiz, Dmitriy Samsonov, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, Jonathan Leane, Talal Aujan, V. Lukas, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Sebastain Graf, Johann-Peter Hartman.

Thank you to all my generous patrons and donaters!
<!-- footer end -->
## Model Details

The GALACTICA models are trained on a large-scale scientific corpus and are designed to perform scientific tasks.
The Alpaca dataset is a set of 52k instruct-response pairs designed to enhace the instruction following capabilites of pre-trained language models.

## Model Use

The GALACTICA model card specifies that the primary indended users of the GALACTICA models are researchers studying language models applied to the scientific domain, and it cautions against production use of GALACTICA without safeguards due to the potential for the model to produce inaccurate information.
The original GALACTICA models are available under a non-commercial CC BY-NC 4.0 license, and the GALPACA model is additionally subject to the [OpenAI Terms of Service](https://openai.com/policies/terms-of-use).