abhinavkulkarni
/

VMware-open-llama-7b-v2-open-instruct-w4-g128-awq

Text Generation

text-generation-inference

Model card Files Files and versions Community

Abhinav Kulkarni commited on Jul 14, 2023

Commit

684f85d

•

1 Parent(s): cf7534e

Updated README

Files changed (1) hide show

README.md +4 -4

README.md CHANGED Viewed

@@ -31,9 +31,9 @@ For Docker users, the `nvcr.io/nvidia/pytorch:23.06-py3` image is runtime v12.1
 ## How to Use
 ```bash
-git clone https://github.com/mit-han-lab/llm-awq \
 && cd llm-awq \
-&& git checkout 71d8e68df78de6c0c817b029a568c064bf22132d \
 && pip install -e . \
 && cd awq/kernels \
 && python setup.py install
@@ -46,7 +46,7 @@ from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
 from accelerate import init_empty_weights, load_checkpoint_and_dispatch
 from huggingface_hub import snapshot_download
-model_name = "VMware/open-llama-7b-v2-open-instruct"
 # Config
 config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
@@ -61,7 +61,7 @@ q_config = {
     "q_group_size": 128,
 }
-load_quant = snapshot_download('abhinavkulkarni/VMware-open-llama-7b-v2-open-instruct-w4-g128-awq')
 with init_empty_weights():
     model = AutoModelForCausalLM.from_config(config=config,

 ## How to Use
 ```bash
+git clone https://github.com/abhinavkulkarni/llm-awq \
 && cd llm-awq \
+&& git checkout e977c5a570c5048b67a45b1eb823b81de02d0d60 \
 && pip install -e . \
 && cd awq/kernels \
 && python setup.py install
 from accelerate import init_empty_weights, load_checkpoint_and_dispatch
 from huggingface_hub import snapshot_download
+model_name = "abhinavkulkarni/VMware-open-llama-7b-v2-open-instruct"
 # Config
 config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
     "q_group_size": 128,
 }
+load_quant = snapshot_download(model_name)
 with init_empty_weights():
     model = AutoModelForCausalLM.from_config(config=config,